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METHODS  AND  TOOLS  FOR  TRAINING  CRISIS  RESPONSE 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  goal  of  this  research  effort  was  to  extend  the  current  U.S.  Anny  Research  Institute 
for  the  Behavioral  and  Social  Sciences  (ARI)  theme-based  training  method  to  understand  how  to 
train  personnel  from  multiple  agencies,  both  military  and  domestic,  who  respond  to  large-scale 
crisis  events.  The  three  research  and  development  initiatives  discussed  in  this  report  advance  the 
initial  Red  Cape  training  method  (Schaefer,  Shadrick,  Beaubien,  &  Crabb,  2008)  by  addressing 
the  training  requirements  of  personnel  who  respond  to  crisis  events. 

Procedure: 

In  order  to  create  the  training  products  that  achieve  the  research  objectives,  a  Cognitive 
Task  Analysis  (CTA)  was  conducted  to  identify  the  cognitive  skills  experts  use  when  responding 
to  emergency  events. 

The  CTA  involved  crisis  management  experts  from  the  military  and  civilian  community, 
including  non- governmental  agencies.  The  interview  data  was  used  to  develop  scenarios  that 
address  the  training  needs  of  diverse  civil-military  organizations  incorporating  the  critical  tasks, 
expert  strategies,  and  decision  requirements  into  the  exercise  scenarios.  In  addition,  the  CTA 
findings  were  used  to  update  behavioral  themes  associated  with  expert  crisis  response.  As  a  part 
of  the  CTA,  faculty  from  the  Garrison  Pre-commander  Course  at  the  U.S.  Anny  Management 
Staff  College  were  interviewed,  and  current  training  was  observed  in  order  to  gain  an 
understanding  of  the  training  needs  and  requirements.  Building  on  previous  training  materials, 
multimedia  training  products  using  animation,  audio,  video,  and  other  methods  were  developed 
to  address  the  training  requirements.  Interview  and  observational  data  were  used  to  create  three 
vignettes  with  facilitated  questions  and  discussion  points  for  garrison  commanders  and  their 
staff.  These  vignettes  focus  on  the  specific  behavioral  themes  and  promote  in-class  discussion 
and  feedback.  A  facilitator  guide  with  questions  and  discussion  points  was  derived  from  the 
interview  data. 

Results  of  the  CTA  were  used  to  develop  a  vignette-authoring  tool  utilizing  the  Think 
Under  Fire  Decisions  (TUF-D)  portal.  To  create  the  tool,  the  manual  scenario  creation  process 
used  in  previous  research  was  adapted  to  develop  a  self-contained,  step-by-step  semi-automated 
process.  An  instructional  overview  guide  was  developed  based  on  systematic  instructional 
systems  design  (ISD)  principles  using  text,  interactive  graphics,  and  videos  to  accommodate  the 
visual  learner  and  audio-based  graphics  for  the  auditory  learner. 
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Findings: 


The  research  and  analysis  resulted  in  three  related  training  products  (theme-based 
training  for  large-scale  crisis  response  involving  multiple  agencies,  an  authoring  tool  for  scenario 
development,  and  crisis  response  training  for  garrison  commanders  and  their  staff).  Interview 
data  revealed  the  individual  knowledge  and  skills  critical  to  effective  decision-making  in  rapidly 
changing  crisis  action  situations  and  differentiated  the  critical  tasks  and  skills  required  for  each 
role  and  agency.  The  data  was  used  to  identify  new  decision  patterns,  refined  the  initial  nine  Red 
Cape  themes,  and  established  a  list  of  vignette  situations.  Three  vignettes  were  created, 
including  scenarios  related  to  an  earthquake,  a  bombing,  and  a  riot  at  a  political  convention.  The 
vignettes  contain  multiple  events  and  branches  that  lead  to  logical  follow-on  events.  This 
training  structure  allows  for  deliberate  practice  with  focused  tasks  and  instant  feedback.  To 
support  vignette  creation,  an  authoring  tools  was  developed  and  integrated  in  the  TUF-D 
simulation  to  allow  a  systematic  vignette  development  and  implementation  process.  An 
instructional  overview  was  produced  to  explain  the  purpose  and  benefits  of  theme-based  training, 
and  provides  instruction  on  how  to  use  the  authoring  tool. 

In  addition  to  the  scenarios  developed  for  large-scale  crisis  response,  three  crisis  action 
vignettes  were  developed  for  garrison  commanders:  a  train  derailment,  a  bombing  attack,  and  a 
tornado.  The  training  vignettes  can  be  completed  over  the  internet  or  on  a  self-contained  CD- 
ROM.  The  goal  of  the  garrison  commander  vignettes  was  to  design  content  that  promoted 
reflectivity,  active  engagement,  and  personal  relevance  of  knowledge.  The  training  comes  with  a 
facilitator  guide  that  is  structured  the  same  for  each  of  the  three  exercises.  A  scene  description 
provides  a  high-level  account  of  the  scenario.  Each  segment  contains  critical  thinking  questions 
and  a  variety  of  discussion  points. 

Utilization  and  Dissemination  of  Findings: 

Anny  National  Guard  units  and  other  federal,  state,  and  local  crisis  response  agencies  can 
use  this  training  program  and  authoring  tool  to  train  emergency  responders  in  their  areas.  They 
can  use  the  authoring  tool  to  create  vignettes  that  address  their  current  and  evolving  training 
needs.  Because  all  TUF-D  products,  including  the  tools  and  scenarios  created  during  this 
research,  are  on  the  National  Guard  server,  they  are  available  at  no  cost  to  all  National  Guard 
units.  The  instructors  at  the  U.S.  Anny  Management  Staff  College  are  using  the  training 
vignettes  to  train  new  garrison  commanders  how  to  respond  to  emergencies  on  Anny 
installations.  The  vignettes  are  tailored  so  that  they  can  be  distributed  and  shared  with  other 
Army  garrison  staffs  to  better  prepare  these  units  to  respond  to  emergency  events. 
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Introduction 

As  tragic  events  such  as  Hurricane  Katrina  have  demonstrated,  a  key  component  of  a 
successful  response  to  a  crisis  event  is  the  ability  of  agencies  to  work  as  coordinated  teams  in 
cooperative  environments.  The  National  Response  Framework  (NRF)  provides  a  scalable, 
flexible,  and  adaptable  framework  for  coordinating  agencies  to  align  key  roles  and 
responsibilities  as  they  manage  incidents  from  the  local  event  up  to  large-scale  terrorist  attacks  or 
catastrophic  natural  disasters  (U.S.  Department  of  Homeland  Security,  2008).  The  national 
framework  calls  for  cooperation  and  understanding  between  local  communities,  tribes,  states,  the 
federal  government,  non-governmental  organizations  (NGO),  and  the  private  sector  as  they  strive 
to  achieve  shared  goals.  Each  responding  agency  is  responsible  for  developing  the  capabilities 
needed  to  respond  to  incidents,  conduct  training  exercises  and  assessments,  provide  resources 
and  capabilities,  and  gather  lessons  learned.  An  effective  response,  as  described  by  the  NRF, 
hinges  upon  invested,  well-trained  leaders  and  responders  who  have  developed  partnerships  and 
are  able  to  achieve  shared  objectives.  The  aim  of  the  research  presented  in  this  document  is  to 
provide  agencies  with  training  tools  and  methods  to  achieve  the  goals  outlined  in  the  NRF. 

The  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  focuses  on  the  “human 
dimension”  of  military  operations  within  future  operational  environments.  TRADOC  discusses 
the  need  for  Soldiers  to  possess  higher  order  cognitive  skills,  such  as  the  ability  to  synthesize 
information  rapidly,  make  quick,  accurate,  situational  assessments,  and  adapt  to  rapidly  changing 
operations  (U.S.  Army  Training  and  Doctrine  Command,  2008).  TRADOC  identified  a  need  for 
training  that  supports  Soldiers  as  they  advance  through  their  careers.  The  training  must 
accommodate  not  only  the  increasing  and  diverse  skill  levels  of  Soldiers  but  also  new  and 
changing  environments  and  situations.  They  call  for  the  development  of  well-designed 
distributed  learning  that: 

•  provides  realistic  training  that  develops  internal  mental  models, 

•  enhances  self-evaluation  and  continual  learning, 

•  contains  virtual  vignettes  and  automated  coaches, 

•  allows  for  tailorable  training  to  fit  specific  needs  and  changing  missions, 

•  enables  Soldiers  to  leam  in  modes  (visual,  auditory,  verbal,  etc.)  that  suit  them  best, 

•  has  the  capacity  to  train  units  without  cumbersome  external  support, 

•  provides  authoring  tools  to  enables  modification  of  scenarios  as  needed,  and 

•  takes  a  systematic  outcome  oriented  approach  to  evaluation. 

Addressing  many  of  the  training  requirements  described  above,  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  conducted  research  to  develop  training 
methods  and  tools  to  train  U.S.  Army  National  Guard  personnel  and  their  civil-military 
interagency  partners  to  respond  effectively  to  crisis  events  (Schaefer,  Shadrick,  Beaubien,  & 
Crabb,  2008).  The  training,  called  Red  Cape:  Crisis  Response  Training  for  National  Guard  and 
Interagency  Teams,  applied  theme-based  training  methodology  to  train  crisis  response  teams. 
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This  report  presents  the  relevant  background  research  that  formed  the  basis  for  the 
original  Red  Cape  training.  In  addition,  the  report  describes  the  methods  and  results  of  research 
conducted  to  expand  the  Red  Cape  methodology  to  address  large-scale  crisis  response,  develop 
methods  and  tools  to  allow  users  to  create  mission  relevant  crisis  response  scenarios,  and 
develop  methods  and  tools  to  train  crisis  response  requirements  for  garrison  commanders  and 
their  staff.  The  purpose  of  the  research  was  to  transition  the  Red  Cape  training  to  a  distributed 
training  environment,  improve  the  existing  training  delivery  technology,  create  a  scenario¬ 
authoring  tool,  and  refine  the  themes  for  national  crisis  response  situations. 

First,  we  discuss  the  need  for  theme-based  training  along  with  the  theoretical  and 
research-based  foundation  upon  which  this  training  was  constructed.  Following  that  discussion, 
we  present  the  research  methods,  development,  and  results  of  the  major  objectives  of  the 
research.  After  discussing  the  research  objectives  and  each  exemplar  training  product,  we 
present  a  short  summary  of  the  entire  training  package  and  then  offer  conclusions  and 
recommendations. 


Justification  for  Theme-Based  Training 

Many  challenges  exist  when  attempting  to  train  teams  fonned  of  diverse  personnel  from 
multiple  agencies  to  cooperate  and  work  together.  Agencies  often  have  different  missions, 
resource  needs  and  requirements,  mission  objectives  and  goals,  and  operating  procedures. 
Training  using  live,  large-scale  exercises  has  been  used  to  train  multiple  agencies  to  respond  to 
an  emergency.  While  live,  multi-agency  emergency  response  training  is  valuable  for  specific 
purposes,  it  has  some  drawbacks,  such  as: 

•  multi-agency  training  rarely  occurs, 

•  multi-agency  training  is  difficult  to  coordinate, 

•  live  exercises  are  costly  in  both  money  and  time, 

•  live  exercises  often  lack  targeted  skills  training,  and 

•  live  exercises  often  do  not  train  to  the  skill  level  of  the  students. 

While  these  factors  inhibit  the  use  of  live  training,  it  is  vital  that  multi-agency  crisis 
training  occurs.  Without  this  training: 

•  responders  do  not  leam  the  needs,  requirements,  and  procedures  of  other  agencies, 

•  turf-wars  erupt  during  events  over  who  is  in  charge, 

•  little  chance  exists  for  cooperative  relationships  to  develop  prior  to  actual  events,  and 

•  conflicting  priorities  and  goals  cannot  be  negotiated  and  resolved. 

Working  in  cooperative  team  environments  during  crisis  events  adds  complexities  to  the 
response  process  that  challenge  most  decision  makers.  Decision-making  in  complex,  high- 
stakes,  emergency  events  requires  the  ability  to  make  sense  of  situations,  sort  though  a  flood  of 
conflicting  infonnation,  manage  uncertainty  and  risk,  and  adjust  actions  and  plans  to  fit  dynamic, 
evolving,  situations  (Klein,  1998).  Engaging  in  this  type  of  cognitive  processing  is  tenned 
“adaptive  thinking.”  The  ability  of  decision  makers  to  adapt  their  thinking  is  particularly 
important  when  plans  do  not  go  as  expected  or  when  facing  novel  challenges.  Adaptive  thinking 
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skills  tend  to  develop  with  experience  and  require  extensive  training  in  realistic  environments 
before  these  skills  will  become  automatic  (Lussier  &  Shadrick,  2004). 

In  large-scale  emergencies,  experienced  decision  makers,  as  well  as  those  with  little  or  no 
experience,  must  engage  in  complex  cognitive  tasks  such  as  making  sense  of  unfolding  events, 
assessing  large  amounts  of  incoming  information,  creating  a  coherent  story  of  current  and  future 
events,  and  determining  the  correct  course  of  action.  Without  knowledge  and  experience  to  draw 
from,  it  is  difficult  to  know  how  to  react  in  an  effective  manner.  Decision  makers  build  this 
knowledge  base  with  experience  that  develops  from  responding  to  real-world  events  or  from 
participating  in  realistic  training  events. 

Research  examining  the  development  of  expertise  demonstrates  that  these  skills  develop 
through  repeated  and  lengthy  exposure  to  situations  requiring  a  specific  skill  set.  This  indicates 
that  individuals  acquire  expertise  in  domain  specific  contexts  and  experts  acquire  skills  by  doing 
tasks  that  promote  learning  in  their  domain  of  specialty  (Ericsson  &  Chamess,  1994;  Ericsson, 
Krampe,  &  Tesch-Romer,  1993).  Theme-based  training  targets  these  experience-borne  skills  by 
identifying  the  cognitive  skills  of  experienced  decision  makers,  categorizing  them  into  themes, 
and  presenting  them  implicitly  during  realistic  scenario-based  training  exercises.  These  themes 
define  the  predominant  cognitive  skills  required  to  handle  a  given  set  of  circumstances  (Schaefer 
et  ah,  2008).  The  themes  for  the  Red  Cape  training  are  listed  below. 

•  Maintain  Focus  on  Mission  Priorities:  Effective  managers  recognize  that  their  primary 
mission  is  to  protect  human  life  and  property,  with  safety  being  paramount. 

•  Keep  the  Chain  of  Command  Flexible:  Effective  managers  recognize  that  the  chain  of 
command  does  not  remain  fixed  throughout  the  crisis. 

•  See  the  Big  Picture:  Effective  managers  remain  aware  of  what  is  happening  around 
them,  with  particular  attention  to  the  impact  on  higher,  adjacent,  lower,  and  supported 
organizations. 

•  Plan  for  and  Recognize  Decision  (Trigger)  Points:  Effective  managers  identify  key  and 
measurable  indicators  that  require  immediate  action. 

•  Reprioritize  as  Necessary:  Effective  managers  continually  reassess  the  scope  and  priority 
of  mission  requirements  as  the  crisis  unfolds. 

•  Use  All  Available  Assets:  Effective  managers  remain  aware  of  what  assets  are  available 
to  them,  what  their  limitations  are,  and  what  lead  times  are  required. 

•  Think  in  Shades  of  Gray,  not  Black  and  White:  Effective  managers  recognize  that  they 
will  be  working  with  imperfect  and  incomplete  information  from  a  wide  range  of 
organizations  and  individuals. 

•  Model  a  Dynamic  Situation:  Effective  managers  recognize  that  the  cause  of  the  incident 
may  be  an  accident  or  act  of  nature,  or  it  may  be  either  criminal  or  terrorist  activity. 

•  Understand  the  Public  Need:  Effective  managers  recognize  that  the  public  will 
experience  a  wide  range  of  emotions,  from  proactive  attempts  at  involvement  to  a  passive 
acceptance  of  the  situation. 

The  knowledge  and  behaviors  encompassed  within  the  themes  stem  not  from  the 
conscious  knowledge  of  experts  but  rather  they  stem  from  the  implicit  skills  developed  through 
experience.  Because  the  themes  provide  a  behavioral  model  of  high-level  cognitive  skills,  they 
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offer  valuable  guidelines  for  training  development.  However,  simply  teaching  the  theme 
definitions  will  not  cultivate  these  cognitive  skills  in  students.  Instead,  students  must  practice 
these  themes  within  the  context  of  relevant  events  and  environments  (Lussier,  Shadrick,  & 
Prevou,  2003). 


Creating  Cognitively  Authentic  Scenarios 

Surrogate  experiences  provide  effective  training  of  high-level  performance  in  crisis 
events.  To  create  surrogate  experiences,  it  is  first  necessary  to  understand  the  perceptual  and 
decision-making  expertise  required  to  handle  situations  with  substantial  cognitive  complexity. 
Then,  it  is  possible  to  create  authentic  and  complex  training  environments  by  incorporating  the 
real  world  experiences  of  experts  (Harris-Thompson,  et  ah,  2004).  This  approach  allows  training 
developers  to  create  scenarios  with  appropriate  context  for  training  cognitive  as  well  as 
procedural  skills. 

Cognitive  Task  Analysis 

Cognitive  Task  Analysis  (CTA)  is  a  set  of  tools  and  techniques  used  to  identify  expert 
behaviors  and  understand  the  critical  tasks  needed  to  create  cognitively  authentic  scenarios.  The 
purpose  is  to  “unpack”  the  knowledge  base  of  domain  experts  so  that  this  knowledge  is  available 
to  training  developers.  Cognitive  Task  Analysis  is  a  family  of  methods  and  tools  used  to  elicit 
the  processes,  cognitive  demands,  and  training  requirements  within  a  specific  task  or  domain. 

The  purpose  of  CTA  is  to  “get  inside  the  heads”  of  the  decision  makers  to  understand  the  factors 
that  guide  their  decision-making  processes  and  identify  other  critical  elements  of  perfonnance 
(Crandall,  Klein,  &  Hoffman,  2006).  The  information  gathered  from  CTA  allows  us  to 
understand  the  cognitive  requirements  that  are  necessary  to  accomplish  the  diverse  challenges 
that  arise  in  multi-agency  settings.  By  documenting  these  cognitive  skills,  the  analyst  is  able  to 
discover  how  decision  makers  think,  clarify  challenging  task  elements,  and  identify  the  support 
needed  to  resolve  crisis  events.  Once  this  knowledge  is  extracted,  it  is  used  to  develop  decision 
support  tools  and  training  programs. 

Deliberate  Practice 

An  understanding  of  critical  cognitive  skills  is  important  to  developing  authentic  training, 
but  the  training  will  not  be  effective  if  it  is  not  delivered  in  a  manner  that  facilitates  learning. 

One  key  to  learning  critical  thinking  skills  is  deliberate  practice.  Deliberate  practice  is  training 
that  focuses  on  expert  behaviors  and  performance  in  difficult  environments  (Lussier  et  ah,  2003). 
Rather  than  requiring  learners,  who  do  not  have  the  necessary  knowledge  base,  to  function  in 
complex  environments,  deliberate  practice  focuses  on  repetitive  training  of  targeted  skills  to  near 
automatic  levels.  When  the  targeted  skills  are  theme-based,  the  skill  development  is  implicit  in 
the  tasks  performed.  For  instance,  training  focused  on  seeing  the  big  picture  might  involve  a 
student  receiving  a  request  from  the  local  mayor  for  assistance  transporting  wounded  citizens  to 
the  hospital.  The  student  is  aware  that  they  are  running  low  on  transport  vehicles.  His  response 
might  be  a  simple  “sorry,  I  can’t  help  you,”  which  insures  he  has  adequate  resources  for  post 
operations.  The  bigger  picture  is  that,  by  not  providing  help,  he  has  fractured  his  relationship 
with  the  local  town  government  and  may  lose  their  support  in  the  future,  or,  by  refusing  to 
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provide  assistance,  off-post  troops  or  family  members  needing  assistance  do  not  get  the 
necessary  care.  In  later  training,  the  student  who  sees  the  big  picture  might  think  creatively  to 
find  the  requested  resources  on  post  or  might  use  his  established  contacts  to  find  resources 
elsewhere.  In  this  way,  the  student  implicitly  learned  to  see  the  big  picture. 

Lussier  et  ah,  (2003)  discussed  the  following  key  characteristics  of  deliberate  practice. 

•  Repetition:  Students  perform  tasks  repetitively  in  order  to  create  automatic  behaviors 
similar  to  experts. 

•  Focused  feedback:  Task  perfonnance  is  evaluated  during  performance  and  feedback 
is  provided  in  the  form  of  positive  and  negative  consequences. 

•  Immediacy  of  performance:  After  feedback,  students  repeat  the  task  in  order  to 
reinforce  the  corrective  actions. 

•  Stop  and  start:  The  feedback  loop  allows  students  to  perform  tasks  in  brief  segments 
with  feedback  breaking  up  the  continuous  flow  of  the  event. 

•  Emphasis  on  difficult  aspects:  With  focus  on  challenging  expert  cognitive  skills,  the 
difficult  rather  than  mundane  (or  procedural)  aspects  of  emergency  response 
situations  are  presented  within  the  scenarios. 

•  Focus  on  areas  of  weakness:  When  instructors  identify  areas  of  weakness,  they  can 
tailor  exercises  to  fit  individual  needs. 

•  Conscious  focus:  Students  consciously  attend  to  the  thinking  behaviors  that  experts 
generally  engage  in  at  an  automatic  level.  By  repetitively  perfonning  these 
behaviors,  students  incorporate  them  into  their  decision  processes  and  they  become 
automatic 

•  Work  vs.  play:  Deliberate  practice  requires  a  level  of  effort  that  is  not  necessarily 
present  in  live  exercises  or  some  game-based  learning  scenarios,  where  training  often 
feels  more  like  play  than  work. 

•  Active  coaching:  Monitoring  students’  perfonnance  should  be  continuous  so  students 
experience  the  implications  of  their  actions  immediately,  as  they  would  in  actual 
events.  They  should  then  leam  corrective  action  and  thus  have  positive  performance 
reinforced  with  positive  consequences. 

Feedback 

A  key  to  deliberate  practice  is  immediate  and  focused  feedback.  Feedback  is  an  essential 
component  of  efficient  decision-making  and  necessary  for  the  development  of  expertise  in  a 
given  domain  (Klein,  1998).  Research  has  found  that  people  who  develop  expertise  tend  to  seek 
out  feedback  after  performing  tasks  (Ericsson  et  ah,  1994;  Ericsson  et  al.,  1993).  Simulated 
exercises  that  incorporate  theme-based  training  provide  instant  feedback  to  students  by 
presenting  the  consequences  of  their  actions  (or  inaction)  and  forcing  them  to  react.  For 
example,  the  consequences  of  not  seeing  the  big  picture  and  refusing  to  assist  the  local  mayor 
might  result  in  negative  media  attention,  reduced  support  in  return,  and  otherwise  preventable 
loss  of  personnel.  These  consequences  may  come  to  the  student  in  the  fonn  of  an  e-mail 
informing  them  of  a  Soldier’s  death  in  the  town  due  to  inability  to  get  to  the  hospital.  Another 
consequence  may  be  a  news  brief  reporting  the  uncooperative  actions  of  the  National  Guard,  or  a 
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telephone  call  from  the  mayor  saying  he  is  ordering  his  volunteer  fire  department  to  discontinue 
their  assistance  in  fighting  a  fire  on  post. 

The  Red  Cape  training  provides  feedback  during  automated  after  action  reviews  (AAR). 
The  AARs  show  the  expert  behaviors  and  categorizes  these  behaviors  by  themes.  In  this  way, 
instructors  can  examine  areas  of  weakness  and  create  scenarios  that  contain  behaviors  and  tasks 
focused  on  the  themes  students  need  to  practice.  For  example,  if  the  AAR  reveals  that  a  student 
rarely  took  into  consideration  the  big  picture,  the  instructor  can  create  a  scenario  that  challenges 
the  student  to  look  beyond  immediate  concerns  and  confronts  the  students  with  multiple 
consequences  when  he/she  takes  action  without  looking  at  the  big  picture.  The  instructor  can 
demonstrate  the  positive  consequences  if  the  student  does  see  the  big  picture.  This  type  of 
automated  and  facilitated  feedback  encourages  students  to  seek  feedback  and  teaches  them  how 
to  self-assess  their  performance — qualities  that  develop  expertise. 

Training  to  Skill  Level 

A  goal  of  deliberate  practice  is  to  shape  automatic  behavior,  thus  it  is  important  to  train  at 
levels  that  are  appropriate  for  the  students  (Shadrick  &  Lussier,  2004).  If  students  confront 
complex  situations  that  are  challenging  to  even  the  most  seasoned  commanders,  they  will 
flounder  and  fail.  Students  need  to  practice  targeted  skills  appropriate  to  their  skill  levels  until 
those  skills  become  automatic.  As  students’  skills  become  automatic  and  the  skill  levels 
increase,  they  can  manage  situations  that  are  more  complex.  Theme-based  training  allows 
instructors  to  initially  create  scenarios  with  few  targeted  theme-based  skills  and  then  increase  the 
level  of  difficulty  as  student  skills  increase.  The  system  allows  instructors  to  present  the 
scenarios  at  a  pace  that  fits  student  skill  levels  and  provides  hints  to  guide  new  students  through 
the  event.  Instructors  can  remove  these  hints  when  students  reach  the  appropriate  level  of 
perfonnance. 

Mental  Models 

By  training  themes,  decision  makers  develop  the  mental  models  necessary  to  manage 
large-scale  events.  Mental  models  are  like  a  picture  puzzle  that  provides  decision  makers  with 
an  accurate  picture  of  the  current  and  future  situation  (Crandall  et  al.,  2006;  Klein,  1998).  The 
more  detailed  and  complete  the  mental  model,  the  clearer  the  picture  and  the  more  detailed  the 
set  of  potential  actions  the  decision  maker  can  access  as  they  map  their  mental  model  onto  the 
actual  situation.  Without  a  relatively  complete  and  diverse  set  of  mental  models,  it  is  difficult  to 
adapt  thinking  to  fit  the  current  situation.  Theme-based  training  improves  perfonnance  by 
focusing  on  the  deliberate  practice  of  these  skills.  This  type  of  practice  provides  decision  makers 
with  the  opportunity  to  fonn  mental  models  and  advance  their  skills  to  levels  closer  to  experts. 
After  receiving  a  small  amount  of  information,  decision  makers  with  incomplete  mental  models 
are  unable  to  fonn  an  accurate  picture  of  the  situation  based  on  the  information  received.  When 
the  mental  model  is  complete,  decision  makers  can  take  those  same  few  bits  of  infonnation  and 
construct  a  near-complete  picture  of  events. 
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Cognitive  Training  for  Teams 

Training  and  performing  as  teams,  particularly  distributed  teams  and  ad  hoc  teams, 
presents  unique  challenges.  While  cognitive  skills  training  often  attempts  to  improve  individual 
skills,  these  improvements  will  not  affect  operations  if  the  team  does  not  function  well  as  a  unit 
at  the  cognitive  level.  If  teams  do  not  have  a  shared  understanding  of  events  or  of  each 
individual’s  role  within  the  team,  they  will  not  function  well  in  team  environments.  Just  as  the 
individual  mind  develops  cognitive  skills,  so  too  does  the  “team  mind.”  A  developed  team  mind 
forms  a  clear  identity,  establishes  routines,  manages  the  flow  of  ideas,  and  engages  in  adaptive 
thinking  (Klein,  1998).  As  the  team  mind  develops,  it  develops  a  stronger  identity,  and  team 
members  are  able  to  monitor  and  assist  one  another,  they  are  better  able  to  manage  uncertainty, 
and  they  more  adeptly  perform  basic  procedures.  Research  on  team  decision  making  revealed 
that  the  key  difficulties  encountered  by  teams  was  that  they  were  not  clear  about  their  roles  and 
functions,  and  that  they  struggled  to  maintain  shared  situational  awareness  (Klinger  &  Klein, 
1999). 


To  maintain  a  shared  situational  awareness,  teams  need  to  calibrate  their  sensemaking. 
Sensemaking  is  the  process  of  understanding  what  is  going  on  in  a  situation  (Weick,  1995).  The 
research  literature  differentiates  between  individual  sensemaking  and  team  sensemaking. 
Individual  sensemaking  occurs  as  people  search  for  connections  between  multiple  events, 
individuals,  and  environments.  Team  Sensemaking  is  team  members’  coordination  of  the  search 
for  these  connections.  Just  as  individual  sensemaking  seeks  to  explain  and  anticipate  events, 
team  sensemaking  coordinates  team  efforts  to  explain  and  anticipate.  Specific  requirements  exist 
during  the  sensemaking  process  at  the  team  level,  such  as  the  need  to  coordinate,  the  ability  to 
maintain  common  ground,  and  the  understanding  to  manage  distributed  knowledge  (Long  & 
Klein,  2006;  see  Figure  1).  Team  sensemaking  involves  the  coordination  of  explanations  and 
anticipations.  A  simple  model  shows  the  major  components  of  team  sensemaking. 


Figure  1.  Team  sensemaking  model. 
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•  Explanations:  It  is  difficult  for  teams  to  combine  data  received  by  various  team 
members.  Team  members  must  know  what  data  to  share  without  overloading  the 
team.  They  need  to  understand  how  their  data  fits  into  the  larger  picture  and  must 
reconcile  their  individual  viewpoint  with  the  team  viewpoint. 

•  Anticipations:  Teams  need  to  direct  and  manage  attention  so  they  can  form  shared 
expectancies  during  the  event.  For  a  team  to  anticipate  events  effectively  they  must 
form  an  integrated  view  of  the  likely  twists  and  turns  that  might  occur  during  the 
situation. 

•  Coordination.  There  are  three  key  components  of  coordination. 

1 .  Controlling  and  improving  the  flow  of  information  -  team  sensemaking  depends 
on  being  able  to  gauge  what  to  share  and  what  to  withhold. 

2.  Evaluating  the  quality  of  data  -  the  team  needs  to  assess  whether  or  not  data  is 
credible. 

3.  Maintaining  and  repairing  common  ground  -  common  ground  is  the  knowledge, 
beliefs,  goals,  commitments,  and  routines  that  permit  effective  communication 
and  coordination.  Common  ground  is  established  by  understanding  what  others 
know,  the  routines  they  can  carry  out,  their  goals,  their  roles  and  functions,  and 
their  current  condition  (e.g.,  workload,  fatigue,  competing  priorities). 

The  themes  incorporate  these  key  components  of  team  sensemaking.  By  training  to  the 
themes,  students  practice  the  requirements  necessary  to  obtain  team  situational  awareness. 
Training  for  teams  should  be  developed  to  allow  the  teams  to  practice  team  sensemaking  and 
begin  to  develop  a  team  mind  when  responding  to  emergency  events. 

In  team  training,  it  is  necessary  to  incorporate: 

•  team  strategies  and  competencies, 

•  the  functions  and  processes  that  teams  need  to  master  (such  as  how  to  communicate 
intent,  or  how  to  assist  other  team  members), 

•  individual  team  member’s  roles  and  the  influence  of  their  roles  on  the  team, 

•  the  team’s  ability  to  develop  a  shared  understanding  of  the  situation,  and 

•  the  team’s  ability  to  monitor  its  own  performance. 

Advantages  of  Theme-Based  Training 

Theme-based  training  provides  instructors  with  a  method  for  training  complex  critical 
thinking  skills.  By  training  to  these  themes,  students  increase  their  ability  to  assess  situations, 
synchronize  personnel,  assets,  and  agencies,  and  execute  plans  by  taking  appropriate  action  in  a 
timely  manner. 


The  following  list  provides  advantages  to  instructors  using  theme -based  training. 

•  Provides  a  method  for  training  implicit  expert  cognitive  skills.  Because  the  cognitive 
skills  are  embedded  within  the  physical  tasks,  instructors  can  choose  the  tasks  to 
teach  rather  than  having  to  figure  out  how  to  teach  complex  thinking  processes. 

•  Provides  focus  within  complex  simulated  events.  Live  events  often  lack  focus  on 
specific  skills.  The  themes  provide  a  guide  for  creating  scenarios  that  provide 
deliberate  practice  of  specific  skill  sets. 

•  Captures  the  complexity  of  multi-agency  response  situations.  The  themes  are 
generalizable  across  agencies  and  responder  roles.  Thus,  the  tasks  specific  to  one  role 
may  not  intersect  with  tasks  required  in  other  roles  yet  they  can  all  train  to  the  same 
theme. 

•  Effective  training  method  evaluated  by  researchers  under  a  variety  of  test  conditions. 
Because  research  has  demonstrated  the  effectiveness  of  theme-based  training, 
instructors  do  not  have  to  wonder  or  assume  the  utility  of  this  training. 

•  Automated  feedback  for  individuals  allows  simultaneous  training  of  multiple 
students.  It  is  difficult  for  instructors  to  monitor  and  give  feedback  to  all  students 
during  complex  simulated  training.  This  training  provides  feedback  based  on  the 
interconnected  theme-based  tasks  to  individual  students. 

•  Themes  developed  based  on  the  experiences  of  experts.  To  create  theme-based 
training,  the  implicit  experience-based  cognitive  processes  of  experts  are  gathered 
using  systematic  knowledge  elicitation  techniques.  Most  training  does  not  extend 
beyond  training  procedures  or  tactics.  Theme-based  training  focuses  on  the  critical 
thinking  skills  that  lead  to  expert  performance. 

•  Behaviors  that  are  linked  to  their  associated  cognitive  skills.  Experts  provide  the 
thoughts  and  decision  processes  associated  with  the  actions  they  took  to  resolve 
actual  crises.  This  creates  a  link  between  behaviors  and  the  thinking  that  lead  to  those 
behaviors. 

•  Trains  at  the  individual  and  team  level.  A  key  to  deliberate  practice  is  training  at  the 
appropriate  level.  As  students’  skills  improve,  instructors  provide  theme -based  tasks 
that  are  increasingly  more  complex;  the  difficulty  of  the  tasks  increase  as  the 
complexity  of  students’  mental  models  increases. 

The  following  list  provides  advantages  to  students  receiving  theme-based  training. 

•  Allows  for  deliberate  practice  of  targeted  skills.  When  a  skill  is  lacking,  a  student 
receives  training  that  focuses  on  their  area  of  weakness.  By  repeatedly  focusing  on 
these  areas,  the  skills  become  automatic  and  are  embedded  within  the  student’s 
mental  model. 

•  Enhances  adaptive  thinking  skills.  In  complex,  uncertain  situations  with  continually 
unfolding  events,  decision  makers  must  adapt  their  thinking  to  the  changing 
circumstances.  Theme-based  training  forces  students  to  engage  in  adaptive  thinking. 
This  type  of  dynamic  thinking  develops  through  extensive  practice.  Students  can 
gain  these  skills  in  controlled  environments  with  consistent  feedback. 
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•  Engaging  training  environment.  Because  theme-based  scenarios  embed  tasks  that 
require  complex  thinking,  they  are  more  engaging  than  simple  scenarios  that  do  not 
resemble  the  real  world. 

•  Development  of  automatic  processes  and  skills.  Repetitive  training  provides  an 
opportunity  to  perform  the  activities  required  which  allows  for  a  deepening 
understanding  of  the  complexities  and  nuances  that  surround  crisis  events.  These 
skills  become  automatic  through  repetition. 

•  Provides  opportunities  to  expand  students’  experience  bases  so  they  are  better 
prepared  to  respond  to  actual  incidents.  With  mental  models  developed  to  a  level  of 
detail  that  resembles  experts,  students  are  better  able  to  respond  during  real  events. 

•  Provides  instant  feedback.  Feedback  is  a  key  to  learning.  When  feedback  occurs 
within  an  event,  students  can  understand  the  consequences  of  particular  actions  and 
learn  how  to  take  corrective  action. 

•  Training  to  appropriate  skill  level.  Training  that  is  focused  on  realism  often  requires 
students  to  perfonn  above  their  current  skill  level.  When  training  is  too  advanced, 
students  become  frustrated  and  do  not  learn  because  they  do  not  have  the  background 
knowledge  and  base  skills  required  by  individuals  at  advanced  levels. 

The  themes  are  general  enough  to  apply  across  a  variety  of  domains,  and  when  thought 
about  in  the  context  of  particular  domains,  they  provide  guidance  about  the  cognitive  and 
behavioral  requirements  specific  to  that  domain.  For  example,  a  theme  when  thinking 
strategically  at  the  command  level  is  “see  the  big  picture.”  This  theme  generalizes  across  many 
situations  and  domains.  When  applied  to  commanders  in  multi-agency  emergency  response 
situations,  examples  of  behavioral  indicators  for  this  theme  might  be  “focus  on  larger  concerns, 
rather  than  deliberating  over  only  small  matters”  or  “take  into  consideration  the  consequences  of 
actions  to  outside  agencies  and  the  community.” 

Summary  of  Projects 

The  next  section  will  discuss  the  exemplar  training  products  developed  during  this 
research.  The  training  methods  and  tools  created  during  this  research  address  issues  to  the  U.S. 
Anny  and  other  emergency  response  agencies.  By  understanding  the  critical  thinking  skills 
necessary  to  perform  during  large-scale  multi-agency  emergencies,  it  is  possible  to  develop 
realistic  training  that  focuses  on  deliberate  practice  and  teaches  adaptive  thinking  at  the 
appropriate  skill  levels.  The  tool  we  use  for  training  delivery  allows  students  to  train  as  teams, 
thus  this  training  promotes  team  sensemaking.  Automated  and  facilitated  feedback  allows 
students  to  develop  the  critical  thinking  skills  necessary  to  gain  expertise  in  this  domain. 

The  report  describes  research  to  develop  training  for  large-scale  multi-agency  emergency 
responses  using  a  theme-based  training  methodology.  After  that  discussion,  we  report  on  a 
vignette  creation  tool  developed  specifically  to  allow  trainers  to  develop  scenarios  that  fit  current 
and  evolving  training  needs.  The  tool  comes  with  a  detailed  instructional  overview  guide  that 
explains  the  theme-based  method  and  the  themes  critical  to  the  actual  training.  The  tool  was 
used  to  create  scenarios  for  large-scale  multi-agency  emergency  response  training.  Finally,  we 
discuss  garrison  commander  training  developed  as  scenario-based,  in-class  training.  The  three 
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scenarios  and  facilitator  questions  in  the  Garrison  Commander  package  are  theme-based  and 
incorporate  the  same  teaching  techniques  as  the  Global  Response  project  (see  Table  1). 

Table  1 


Project  Outcomes 


Global  Response 
Training 

Authoring  Tool 

Garrison  Commander 
Training 

> 

Updated  Red  Cape  themes 

>  Scenario  authoring  tool 

>  In-class  theme-based 

> 

Three  CTA  driven 

>  Instructional  overview 

training 

scenarios 

guide 

>  Three  CTA  driven  scenarios 

> 

Web-based  training 

>  Facilitator  guide 

exercises 

Global  Response:  Large  Scale  Interagency  Training 

The  Red  Cape:  Crisis  Action  Planning  training  program  centered  on  training  crisis 
response  skills  to  Anny,  Army  National  Guard,  civil-military,  and  other  agency  partners  but  did 
not  extend  to  multiple  agencies  responding  at  a  national  level  (Schaefer  et  ah,  2008).  In  addition, 
Red  Cape  training  requires  students  to  assemble  in  a  single  location,  which  does  not  provide  a 
suitable  format  for  training  at  a  national  level.  Thus,  Red  Cape  was  extended  to  address  these 
needs  by  creating  a  distributed  web-based  training  program  and  by  evaluating  and  expanding 
Red  Cape  training  to  address  issues  relevant  to  multiple  agencies  responding  to  situations  at  the 
national  level. 

The  main  objectives  of  the  Global  Response  research  were  to  refine  and  expand  the 
existing  Red  Cape  expert  themes  and  to  use  existing  technology  to  facilitate  online,  collaborative 
training  with  partners  from  diverse  agencies  and  nations.  Our  purpose  was  to  train  leaders  to  be 
agile  and  adaptive  in  national  crisis  action  situations,  to  plan  for  homeland  emergency 
contingencies,  and  to  support  civil  authorities  while  simultaneously  reducing  the  collaborative 
barriers  between  the  Anny  and  civil  authorities.  Another  purpose  was  to  reduce  the  cultural 
barriers  between  the  Army  and  emergency  first  responders,  allowing  for  exchange  of  standing 
operating  procedures  (SOP)  and  exposure  to  differences  in  agency  language  and  communication. 
Our  approach  was  to  develop  authentic  scenarios  that  emulate  the  situational  factors  experts 
perceive  within  the  emergency  response  domain  in  order  to  foster  within  students  the  perceptual 
and  decision-making  processes  of  experts  (Ross  &  Pierce,  2000).  Using  computer-based,  event 
driven  training  allows  instructors  to  create  scenarios  that  target  specific  skills  and  are  appropriate 
to  the  level  of  the  learner.  Students  from  multiple  agencies  can  train  simultaneously  at  any  time 
from  any  place  with  web  access.  This  allows  agencies  in  various  locations  to  train  together 
without  travel  or  other  expenses. 

The  previous  Red  Cape  training  presented  scenarios  using  static  slide  presentations  and 
Flash®  animation.  This  updated  version  or  Red  Cape  engages  students  in  active  vignettes  that 
simulate  the  fast  pace,  infonnation  overload,  and  time-pressure  of  real-world  events.  Students 
can  train  individually  with  simulated  players  or  as  a  group.  Individually,  students  leam  the  tasks 
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they  are  responsible  for,  how  information  flows  in  a  dynamic  environment,  and  how  their  actions 
influence  other  agencies.  In  a  multi-player  environment,  students  must  communicate  with  other 
students  occupying  diverse  roles  within  multiple  agencies.  This  provides  opportunities  for 
students  to  interact  and  communicate  with  students  from  other  agencies  and  discover  the 
requirements  and  procedures  of  those  agencies. 

The  Think  Under  Fire  Decisions  (TUF-D)  training  program,  developed  for  the  National 
Guard  Bureau,  was  used  to  create  an  environment  for  presenting  training  scenarios.  The  TUF-D 
is  a  collaborative,  distributed,  web-based  training  program  that  presents  scenarios  to  command- 
level  responders  and  allows  users  to  train  as  individuals  or  as  teams  (see  Appendix  A).  The 
system  is  the  follow-on  to  the  Automated  Exercise  and  Assessment  System  (AEAS)  developed 
to  deliver  simulated  multi-role  exercises  in  a  networked  environment.  The  system  was  modified 
to  support  current  multi-agency  command-level  decisions  in  emergency  operation  situations. 

The  TUF-D  enhances  Red  Cape  training  by  allowing  personnel  from  multiple  organizations  or 
agencies  to  train  together  asynchronously  and/or  synchronously  in  a  common  scenario. 

This  system  trains  individuals  at  their  appropriate  skill  level  by  focusing  on  the  three 
phases  of  learning  defined  by  Bloom’s  taxonomy:  knowledge,  skills,  and  abilities  (Bloom, 

1956).  Each  phase  provides  the  user  with  a  different  learning  experience  but  draws  from  a  single 
knowledge  database  to  ensure  consistent  training. 

•  In  the  Knowledge  phase,  students  are  exposed  to  accepted  doctrine  in  the  domain. 

This  exposure  teaches  students  the  basic  concepts  and  shared  vocabulary  of  the 
domain.  Learning  is  accomplished  through  classic  presentation-based  training. 

•  In  the  Skills  phase,  students  transform  knowledge  into  behavioral  demonstrations  of 
the  material.  Students  start  to  build  a  repertoire  of  experiences  that  form  mental 
models.  Learning  is  accomplished  through  individual  computer-based  simulation.  In 
TUF-D,  individual  students  interact  with  simulated  team  members.  Students  may  try 
different  roles  within  the  simulation  to  gain  an  understanding  of  the  responsibilities  of 
their  team  members. 

•  In  the  Abilities  phase,  students  execute  their  roles  as  a  team  in  a  collective 
simulation-based  exercise.  Learning  in  this  phase  is  accomplished  through  team 
computer-based  simulation.  In  TUF-D,  individual  students  perform  tasks  that 
influence  other  team  members  and  must  react  to  the  actions  of  other  live  players. 

Team  exercises  teach  students  about  each  other’s  roles,  abilities,  responsibilities,  and 
lines  of  communication. 

One  advantage  of  this  training  is  that  it  helps  students  build  a  pattern  base  or  mental 
model  to  make  better  recognitional  decisions  when  real  world  challenges  arise.  The  system 
assesses  all  phases  of  learning  by  drawing  from  a  single  knowledge  base,  which  provides 
consistent  training  across  multiple  presentation  methods  and  levels  of  experience  (Pigora, 
Tamash,  &  Baxter,  2006). 
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Method 


Participants 

Participants  were  29  active  duty  Anny  personnel  who  had  experience  responding  to 
national,  multi-agency  crisis  events.  They  participated  in  interviews  and  two  focus  groups.  In 
addition,  14  National  Guard  personnel  from  Camp  Atterbury,  Columbus,  IN,  and  12  personnel 
from  the  Indiana  National  Guard,  Indianapolis,  IN  participated  in  interviews  and  two  focus 
groups.  Participants,  including  a  Joint  Task  Force  Commander  during  a  hurricane  federal 
response  effort,  staff  officers  in  Northern  Command,  staff  members  for  the  Defense  Coordinating 
Element,  Region  VII,  and  members  of  the  Indiana  Homeland  Security  and  Army  National  Guard, 
participated  in  telephone  interviews. 


Procedure 

The  research  built  on  the  Red  Cape  research  program  by  interviewing  crisis  management 
experts  from  the  civilian  and  military  community  about  the  ability  of  military,  civilian,  and  non¬ 
governmental  agencies  to  work  together  in  response  to  large-scale,  crisis  action  situations.  The 
interviewers  utilized  an  interview  guide  based  on  CTA  methodologies  with  a  focus  on  the  critical 
decisions  and  tasks  present  during  multi-agency  emergencies.  Subject-matter  experts  (SME) 
were  interviewed  about  specific  multi-agency  incidents  in  which  they  faced  decision  challenges 
and  struggled  to  understand  and/or  resolve  crises.  Interviews  also  focused  on  identifying  training 
issues  relevant  to  crisis  action  planning  and  execution  including  joint,  national  missions  (see 
Appendix  B).  The  goal  during  these  interviews  was  to  identify  the  critical  cues  and  factors 
necessary  during  the  sensemaking  processes  and  the  essential  tasks  that  contribute  to  effective 
response  during  crisis  events.  The  information  gleaned  during  the  interviews  was  invaluable 
during  the  development  of  scenarios  that  addressed  the  training  needs  of  diverse  civil-military 
organizations. 

In  addition  to  conducting  CTA  interviews  in  Indiana,  SOPs  and  other  materials  were 
collected  in  order  to  understand  operating  process  and  to  develop  realistic  vignettes.  From  the 
CTA  data  and  other  documentation,  the  research  team  developed  exemplar  scenarios  by 
constructing  story  lines,  identifying  multiple  decision  points,  and  incorporating  the  multiple 
strategies  used  to  resolve  each  event.  The  three  scenarios  were  an  earthquake,  a  bombing,  and  a 
riot  at  a  political  convention.  The  infonnation  was  incorporated  into  scenarios  as  injects  (key 
events  to  stimulate  an  action  or  response).  Each  inject  has  associated  consequences  (multiple 
branches)  that  lead  to  logical  follow-on  events.  This  training  structure  allows  for  deliberate 
practice  with  focused  tasks  and  instant  feedback. 

Analysis 

Data  from  each  interview  was  categorized  by  matching  infonnation  gathered  from  the 
participants  to  the  definitions  of  the  original  Red  Cape  themes.  In  addition,  categories  for 
scenario  and  inject  ideas,  explanations  of  agency-specific  procedures/roles,  and  training 
suggestions  for  use  in  scenario  development  were  documented.  Spreadsheets  were  utilized  to 
categorize  the  interview  data  into  each  theme  or  category;  each  row  contained  data  from  one 
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interview.  Placing  each  piece  of  information  into  the  appropriate  cell  by  matching  it  with  a 
theme  or  category  definition  allowed  researchers  to  gauge  the  amount  of  infonnation  in  each 
category,  by  participant  and  by  comment.  The  categorized  information  resulted  decision 
requirements  tables  (DRT).  The  DRT  categories  were  Decision  Point,  Why  Difficult, 
Cues/Factors,  Strategies,  and  Novice  Errors. 

For  example,  the  list  below  contains  the  categorized  interview  data  for  the  original  Red 
Cape  theme  See  the  Big  Picture. 

1 .  Some  people  are  not  capable  of  seeing  the  big  picture. 

2.  Use  geographic  infonnation  system  (GIS)  to  get  a  big  picture  view. 

3.  New  commanders  do  not  understand  the  system  so  they  cannot  see  the  big  picture. 

4.  Think  about  the  impact  of  your  actions  beyond  the  immediate  threat. 

5.  Can  create  a  bigger  problem  by  trying  to  solve  the  problem. 

6.  Cannot  just  act  without  considering  public  reaction. 

7.  It  is  easy  to  be  overwhelmed  by  events  and  get  tunnel  vision. 

8.  Worrying  about  small  things  during  a  large  crisis  is  not  seeing  the  big  picture. 

9.  It  is  difficult  to  see  past  observable  facts  and  piece  together  cues. 

10.  Make  sure  people  know  your  weaknesses  and  infonnation  needs. 

The  data  was  then  mapped  onto  a  DRT  under  the  updated  theme  see  the  big  picture  and 
predict  events  (see  Table  2). 

Table  2 


Decision  Requirement  Table  of  Data  Categorized  Under  the  Theme  “ See  the  Big  Picture  ” 


Problem/Decision 

Why  Difficult? 

Cues/Factors 

Strategies 

Novice  Errors 

Maintain  big  picture 

5.  Can  create  a  bigger 

8.  Worrying  about 

2.  Use  GIS  to  get 

view 

problem  by  trying 
to  solve  the  current 
issue 

6.  Can’t  just  act 
without  considering 
public  reaction 

7.  It  is  easy  to  be 
overwhelmed  by 
events  and  get 
tunnel  vision 

9.  It  is  difficult  to  see 
past  observable 
facts  and  piece 
together  cues 

small  things 
during  a  large 
crisis,  is  not 
seeing  the  big 
picture 

a  big  picture 
view 

4.  Think  about 
the  impact  of 
your  actions 
beyond  the 
immediate 
threat 

Know  people’s 

1 .  Some  people  are 

10.  Make  sure 

3.  New 

strengths  and 

not  capable  of 

people  know 

commanders 

weaknesses 

seeing  the  big 
picture 

your 

weaknesses  and 

information 

needs 

do  not 

understand  the 
system  so  they 
cannot  see  the 
big  picture 
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Results  and  Discussion 


The  interview  data  revealed  the  individual  knowledge  and  skills  critical  to  effective 
decision-making  in  rapidly  changing  crisis  action  situations  and  differentiated  the  critical  tasks 
and  skills  required  for  each  role  and  agency.  From  these  interviews,  new  decision  patterns  were 
identified,  the  initial  nine  Red  Cape  themes  were  refined,  and  a  list  of  vignette  situations  was 
created. 

The  categorized  information  generally  fit  the  original  Red  Cape  themes,  however,  close 
analysis  indicated  that  clarification  and  reorganization  of  some  themes  was  necessary.  Based  on 
the  original  definitions,  some  themes  such  as  think  in  shades  of  gray  and  model  a  dynamic 
situation,  had  very  little  information  categorized  into  them  from  the  interviews,  thus  these 
themes  were  consolidated  into  other  themes.  The  data  drove  the  fonnation  of  three  new 
categories:  communication,  coordination/synchronization,  and  sensemaking/information 
gathering.  All  the  existing  and  new  themes  described  one  of  three  different  cognitive  processes: 
Assess,  Synchronize,  and  Execute.  These  processes  involve  activities  that  cross  over  all  themes, 
such  as  leveraging  established  relationships  and  establishing  a  battle  rhythm.  They  serve  to  link 
together  the  themes  into  an  evolving  response  pattern  of  interdependent  actions  characterized 
within  the  themes  (for  detailed  process  and  theme  definitions  see  Appendix  C). 

Updated  Themes 

Below  is  a  list  of  the  update  Red  Cape  themes  for  each  process  (Assess,  Synchronize,  and 
Execute). 


•  Assess: 

Collect  and  assess  information  and  understand  the  situation:  Effective 
commanders  and  staff  are  able  to  detennine  quickly  how  and  where  to  collect 
information  necessary  to  develop  a  situational  understanding. 

See  the  big  picture  and  predict  events:  Effective  commanders  and  staff  construct 
a  coherent  picture  of  unfolding  events  and  see  the  overarching  implications  of 
possible  actions. 

•  Synchronize: 

Coordinate  and  communicate  internally  and  externally  across  all  agencies: 
Effective  commanders  and  staff  collaborate  within  and  across  agencies  in  a 
manner  that  develops  a  shared  understanding  of  the  mission  and  of  responder 
roles,  while  enabling  the  synchronization  of  mission  goals  and  objectives. 

Acquire,  prioritize  and  allocate  assets:  Effective  commanders  and  staff  determine 
the  type  and  amount  of  equipment  and  personnel  necessary  to  accomplish  the 
mission. 

Establish  chain-of-command:  Effective  commanders  and  staff  make  clear  to  all 
personnel  who  is  in  charge  of  an  incident  and  which  staff  or  agencies  are 
responsible  for  various  actions. 

•  Execute: 

Recognize  decision  points:  Effective  commanders  and  staff  recognize  the  point  in 
a  crisis  when  they  need  to  make  a  decision. 
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Address  public  and  responder  needs:  Effective  commanders  and  staff  notify  the 
public  and  media  about  the  situation  and  relevant  events. 

Maintain  mission  priorities:  Effective  commanders  and  staff  recognize  that  their 
primary  mission  is  to  protect  human  life  and  property,  while  insuring  the  safety  of 
incident  responders. 

Table  3  shows  the  match  between  the  updated  and  original  themes.  New  themes  were 
created  to  account  for  the  data  and  combined  some  original  themes  to  reflect  one  updated  theme. 
The  definitions  for  the  updated  themes  incorporated  all  the  original  themes  and  their  associated 
cognitive  skills. 

The  interviews  revealed  several  observable  behaviors  associated  with  each  theme.  For 
instance,  someone  who  is  seeing  the  big  picture  and  predicting  events  will  display  behavioral 
indicators  such  as: 

•  focusing  on  larger  concerns,  rather  than  deliberating  over  only  small  matters, 

•  taking  into  consideration  the  consequences  of  their  actions  to  outside  agencies  and  the 
community, 

•  making  interpretations  beyond  observable  facts, 

•  creating  a  story  from  small  bits  of  information, 

•  focusing  on  more  than  just  one  aspect  of  the  situation, 

•  taking  effective  action  when  new  infonnation  comes  in,  and 

•  setting  a  clear  battle  rhythm  and  maintaining  motivation. 


Table  3 


Comparison  of  Updated  Themes  to  Original  Themes 


Updated  Themes 

Original  Themes 

Assess 

•  Collect  and  assess  information  and 
understand  the  situation 

•  See  the  big  picture  and  predict  events 

•  See  the  big  picture 

•  Think  in  shades  of  gray,  not  black  & 

white 

Synchronize 

•  Coordinate  and  communicate  internally 
and  externally  across  all  agencies 

•  Acquire,  prioritize  and  allocate  assets 

•  Use  all  available  assets 

•  Establish  chain-of-command 

•  Keep  the  chain-of-command  flexible 

Execute 

•  Recognize  decision  points 

•  Plan  for  and  recognize  decision  points 

•  Reprioritize  as  necessary 

•  Model  a  dynamic  situation 

•  Address  public  and  responder  needs 

•  Understand  the  public  need 

•  Maintain  mission  priorities 

•  Maintain  focus  on  mission  priorities 
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The  behavioral  indicators  were  extracted  from  the  DRT  data  (see  Table  2).  Data 
categorized  into  the  Why  Difficult  section,  such  as  “it  is  difficult  to  see  past  observable  facts  and 
piece  together  cues”  indicates  that  someone  who  is  making  interpretations  beyond  observable 
facts  is  displaying  the  behaviors  defined  in  the  see  the  big  picture  theme.  Observable  behaviors 
might  be  a  staff  member  who  receives  a  small  bit  of  information,  but  takes  action  that  indicates 
he/she  has  a  broader  interpretation  of  that  infonnation.  For  example,  one  participant  recounted 
an  incident  when  he  was  looking  into  a  suspicious  truck  and  noticed  wires  sticking  out  from 
under  the  seat.  He  instantly  backed  up,  evacuated  the  area,  and  called  a  bomb  squad.  He  took  a 
small  amount  of  infonnation  (the  wires)  and  made  an  interpretation  beyond  observable  facts  (a 
bomb).  (See  Appendix  C  for  the  behaviors  associated  with  each  theme). 

Scenario  Creation 

Scenarios  tailored  to  emergency  response  at  a  state-level  National  Guard  Joint  Operations 
Center  (JOC)  were  created  by  combining  input  from  government,  military,  and  support 
personnel.  The  vignettes  were  developed  using  the  Indiana  National  Guard  JOC  as  a  setting 
because  of  their  ongoing  sponsorship  during  this  research  and  because  of  their  close  proximity  to 
the  research  team.  After  identifying  the  themes,  expert  behaviors  for  each  scenario  were 
identified  using  subject-matter  experts  from  the  Indiana  National  Guard  and  their  civil-military, 
interagency  partners.  An  instructional  training  approach  was  developed  that  required  students  to 
apply  the  themes  within  a  simulated  event.  To  create  scenarios  that  fit  the  cognitive  and  physical 
skills  of  expert  responders,  the  following  scenario  development  requirements  were  considered. 

•  Scenarios  must  exercise  the  skills  required  to  meet  training  objectives. 

•  Scenarios  must  be  realistic.  If  not,  students  will  not  be  motivated,  and  it  will  be 
difficult  to  transfer  skills  (i.e.,  from  a  simulation  into  real-world  situations).  Realistic 
scenarios  include  probable  real-world  events,  realistic  duties,  tasks,  and  interactions 
with  teammates,  along  with  believable  documents,  news  reports,  and  orders. 

•  Scenarios  must  match  students’  background  knowledge  base.  Because  training  is 
usually  conducted  under  time  constraints,  students  should  spend  minimal  time 
learning  about  the  background  scenario  and  spend  maximum  time  applying  their 
knowledge  (first  phase  of  Bloom’s  Taxonomy)  to  events  in  the  scenarios.  Various 
strategies  exist  to  minimize  background  knowledge  across  scenarios: 

-  keep  storyline  structure  and  background  knowledge  constant  across  scenarios, 
create  a  realistic,  but  simplified,  set  of  situations,  and 
rely  on  known,  existing  situations  to  create  scenarios. 

The  scenario  concepts  were  derived  from  incidents  suggested  during  the  CTA  interviews. 
The  three  general  scenario  storylines  were  an  earthquake,  a  bombing,  and  a  riot  at  a  political 
convention.  The  storylines  were  selected,  in  part,  because  participants  expressed  a  need  for 
training  to  respond  to  uncommon  natural  disasters  such  as  earthquakes,  more  experience 
handling  terrorist  events,  and  more  experience  responding  to  atypical,  unplanned  events,  such  as 
a  riot.  Critical  crisis  response  skills  were  extracted  from  the  data  and  a  strategy  was  developed  to 
detennine  how  best  to  represent  the  skills  in  the  scenarios.  During  either  face-to-face  meetings 
or  teleconferences,  ideas  were  identified  for  each  vignette.  Each  vignette  was  scripted  in 
storyboard  format  and  appropriate  maps,  documents,  photos,  information,  overlays,  and  visual 
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cues  relevant  to  that  event  were  identified.  After  storyboarding  the  scenarios,  SMEs  provided 
key  considerations  and  possible  solutions  to  the  challenges  presented  in  the  scenarios  and 
described  the  actions  they  might  take  to  resolve  the  situation.  In-house  and  external  personnel 
reviewed  vignette  drafts  and  provided  feedback.  The  review  process  was  completed  several 
times  before  completing  the  final  product. 

Creating  Training  in  TUF-D 

The  existing  TUF-D  training  tool  was  used  to  create  the  training  based  on  the  Global 
Response  research  objectives.  The  TUF-D  is  a  simulation-based  training  system  that  allows 
individuals  or  teams  to  make  decisions  in  realistic  situations  and  experience  the  outcomes.  It 
immerses  users  in  scenario-based  exercises  by  presenting  e-mail,  simulated  telephone  calls, 
website  news  stories,  video  clips,  and  other  multimedia  communications  that  resemble  current 
environments.  The  TUF-D  is  completely  web-based  with  no  software  install.  Users  make 
decisions  and  mitigate  incidents  by  responding  to  messages,  communicating  with  teammates, 
and  giving  commands  to  simulated  entities. 

The  TUF-D  was  designed  with  an  interface  familiar  to  users  in  order  to  minimize  the 
training  necessary  to  use  the  system  (Figure  2).  By  creating  an  interface  that  resembles  a 
standard  Windows  desktop,  including  realistic  e-mail,  a  recognizable  instant  messenger  program, 
and  familiar  documents,  specialized  interface  instruction  was  reduced.  In  addition,  the  TUF-D 
map  display  uses  the  Google  Maps  interface  with  scenario  icon  overlays  to  show  populations, 
resource  positioning,  facilities,  and  critical  information.  The  user  may  also  choose  to  receive 
multimedia  inputs  as  video,  with  or  without  closed-captioning.  To  further  assist  users,  a  user- 
guide  with  a  step-by-step  tutorial  on  how  to  set  up  and  run  a  training  session  was  developed. 

The  guide  provided  users  with  a  help  option  during  the  simulation.  The  user  guide  also  provides 
definitions  of  the  themes  and  an  explanation  of  how  and  why  theme-based  training  is  useful  for 
building  the  mental  models  necessary  to  excel  in  crisis. 

The  TUF-D  development  for  this  project  encompassed  two  new  areas  of  functionality: 
Voice  over  Internet  Protocol  (VoIP)  communications  during  the  exercise,  and  a  Scenario 
Authoring  Tool.  The  Authoring  Tool  is  discussed  in  the  “Implementing  Unique  User 
Requirements  and  Transitioning  Red  Cape  Training”  section  of  this  report. 
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Figure  2.  TUF-D  interface. 


VoIP 

One  goal  of  this  research  was  to  provide  VoIP  communications  to  participants  in 
distributed  locations.  The  VoIP  provides  several  advantages  for  the  TUF-D  system  as  a  whole, 
thus  our  requirements  analysis  included  both  requirements  addressed  in  this  research  and  some 
requirements  that  are  beyond  the  scope  of  this  effort.  The  requirements  for  the  current  researcher 
were: 

•  Interface: 

connect  two  players  in  a  simulated  “telephone  call”  during  a  TUF-D  exercise, 
connection  requires  no  more  interface  overhead  than  does  current  simulated 
telephone  calls  with  simulated  players,  and 
ability  to  pick  a  telephone  call  topic  for  use  in  task  assessment. 

•  Software: 

ability  to  integrate  with  a  Flash-based  interface, 
ability  to  support  point-to-point  connections, 
no  client  install,  and 

-  use  of  computer  speakers  and  microphone  for  capturing  and  playing  audio. 

The  requirements  for  future  research  were: 

•  Software: 

ability  to  support  “conference  call”  connections, 
record  audio  (server  side)  for  After  Action  Review,  and 
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ability  to  add  server-side  plug-ins  to  process  data  (e.g.,  support  speech  recognition 
for  task  assessment). 

Several  VoIP  server  options  were  surveyed,  including:  Adobe  Flash  Interactive  Server, 
Wowza,  Mumble/Munnur,  FreeSwitch,  Red5,  and  Adobe  Pacifica.  The  Adobe  products  and 
Wowza  were  capable  but  had  licensing  fees.  Red5  encompassed  all  the  functionality  needed  in 
an  open  source  server.  Because  Red5  was  the  server  previously  selected  for  TUF-D  video 
streaming,  it  was  easy  to  add  the  VoIP  support  to  our  existing  video  server  setup. 

Performance  Assessment 

The  training  system  provides  ongoing  feedback  during  training  and  an  after-action  review 
at  the  end  of  the  simulation.  Assessment  of  learner  performance  occurs  through  four 
mechanisms. 

1.  Knowledge  Trainer.  To  enhance  basic  knowledge,  the  training  contains  a  built-in  web- 
based  training  presentation  system  that  presents  content  and  quizzes  students. 

2.  Direct  Feedback.  Simulated  and/or  live  role  players  provide  timely  and  contextual 
feedback  to  students  during  the  training  scenario.  This  feedback  comes  in  the  form  of 
infonnation  and  requests  that  reflect  the  consequences  of  students’  previous  actions.  This 
feedback  requires  students  to  take  action  to  remedy  the  situation. 

3.  Posted  Questions.  Facilitators  can  post  direct  questions  to  students.  They  can  ask 
questions  at  any  time  during  the  scenarios  in  order  to  measure  students’  situational 
awareness,  identify  their  decision  processes,  and  provide  feedback  about  actions  taken. 

4.  After  Action  Report.  Upon  completion  of  the  training  scenario,  the  students  and 
facilitators  receive  detailed  after  action  reports  (AAR)  containing  all  of  the  actions 
taken/not  taken  during  the  simulation  (Figure  3).  The  AAR  outlines  student 
accomplishments  and  provides  feedback  about  their  overall  performance  within  the 
training.  Students  and  facilitators  can  sort  the  AARs  by  time  or  by  theme.  From  this 
information,  instructors  can  evaluate  which  tasks  students  performed  well  and  which 
tasks  they  showed  weak  performance.  This  provides  insight  into  the  themes  that  require 
extra  or  focused  training.  Students  receive  their  own  AARs  based  on  individual  actions 
and  decisions.  Exercise  administrators  can  view  separate  AARs  for  each  role. 
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TUF-D 


Think  Under  Fire  Decisions 

After- Action  Review 


A  After  Action  Review 


Task:  Handle  a  missile  launch 


Determine  validity  of  launch  reports 


e  [Completed  by  testpl 


increase  defensrve  posture  in  response  to  missle  launch 


:  ompieted  by  testpl 


s ; 


ancel  today's  training  in  response  to  missle  launch 
ompieted  by  testpl 


Schedule  staff  meeting 

►  Note  Time  ran  out  to  do  this  task 

J  Should  have  been  completed  by  Joint  Forces  Air  Component  | 
Commander 


Task:  Prepare  Intelligence  for  the  Joint  Operations  Area  (JOA) 


Submit  CFC  Brief 
Note  A  reminder  was  sent 

To  be  completed  by  Joint  Forces  Air  Component  Commander 


1>  Checkable 


Confirmation  of  Recent  Missile  Launch  from 
Northland 
To: 

I  just  saw  a  news  flash  that  stated  that  Northland  has  launched 
another  missile  Can  you  confirm  these  reports'’ 

From: 

Yes  Sir.  It  has  been  confirmed  that  Northland  has  launched  another 
test  missile  from  Kumar  The  missile  was  launched  at  approximately 
04  20  hours  and  just  landed  m  the  Ocean  26  miles  off  of  Southland's 
Northeast  coast  The  missile  was  identified  as  a  Zimmerman  II  and 
has  a  range  of  1500km  with  an  impact  radius  of  600  meters  This 

nd  claims  that 


A,  Task  Details 


Triggered  by: 

Dhefmg  message 


BBS 


^ompieted  at: 

Completion  by: 
Consequence: 


Not  Completed 
Nov  27  0440 


|no  consequences  or  the  task  was  not  completed 

Reminder  Sent:  |  Rtmindm 


Figure  3.  After  Action  Review. 

Evaluation  of  Scenarios/Training 

In  order  to  determine  the  effectiveness  of  the  scenario  creation  tool  and  the  training 
program,  an  evaluation  of  both  the  system  and  the  training  content  was  conducted. 

Method 


Participants.  Twelve  National  Guard  members  with  experience  in  Emergency 
Operations  Center  participated  in  the  evaluation.  Members  of  the  staff  included  Soldiers  and 
civilians  from  the  various  response  cells  including  the  G-3,  EOC  Director,  G-4,  Provost 
Marshall,  Safety  Officer,  and  other  staff  agencies.  The  same  participants  attended  two  days  of 
evaluation. 


Procedures.  The  evaluation  occurred  during  two  3 -hour  sessions  held  on  separate  days. 
These  sessions  included  instruction  on  and  practice  with  training  system.  Activities  on  Day  1 
included  an  overview  of  the  themes  and  the  scenario  learning  objectives,  instruction  on  the  use  of 
the  training  simulation  program,  participation  in  a  simulation  session,  questionnaires  and 
discussion.  Activities  for  Day  2  included  an  overview  and  instruction  of  the  scenario  creation 
tool,  guided  practice  using  the  tool,  user  creation  of  scenarios  using  the  tool,  and  questionnaires 
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(further  discussion  of  Day  2  takes  place  in  the  “Implementing  Unique  User  Requirements  and 
Transitioning  Red  Cape  Training”  section). 

Prior  to  starting  the  session  on  Day  1 ,  all  participants  read  and  signed  informed  consent 
forms.  Participants  positioned  themselves  where  they  could  see  the  screen  of  one  of  the 
government  furnished  laptops  used  in  this  evaluation.  Participants  were  introduced  to  the 
research  objectives  and  provided  with  a  detailed  explanation  of  the  purpose  of  the  evaluation 
activities.  Participants  then  received  an  explanation  of  theme-based  training  and  a  quick  briefing 
on  how  to  operate  the  training  system.  After  a  30-minute  hands-on  training  session,  the  research 
team  observed  participants  as  they  completed  the  earthquake  scenario  exercise. 

During  the  exercise,  the  TUF-D  encountered  some  technical  problems  that  interrupted  the 
session  until  the  problem  was  resolved.  The  primary  problems  included  intermittent  VoIP, 
particular  features  not  operating,  and  instances  when  the  system  locked-up.  The  technical 
problems  were  attributed  to  server  updates  that  were  being  completed  at  the  time  of  the  exercise 
during  an  iterative  development  schedule. 

Results  and  Discussion 

The  exercise  presented  issues  and  problems  to  participants  that  were  well  above  their 
usual  scope  of  responsibility.  Even  with  the  technical  problems  identified  above,  the  research 
team  was  able  to  observe  the  participants  operate  the  system  and  draw  conclusions  about  its  use 
in  training.  Results  from  this  evaluation  include  participant  comments,  observations  made  by  the 
research  staff,  and  data  from  questionnaires  about  the  training. 

The  following  comments  about  the  training  tool  represent  those  made  by  the  evaluation 
participants. 

•  Participants  generally  liked  the  features  of  the  training  tool  and  expressed  interest  in 
using  it  to  train  in  the  future.  They  commented  that  EOC  response  teams  normally  do 
not  have  the  opportunity  to  train  or  work  together  until  a  real-world  crisis  occurs. 
They  thought  they  would  use  the  system  to  do  distributed  desktop  exercises  in  order 
to  improve  ability  to  work  together. 

•  The  participants  particularly  liked  the  e-mail  and  telephone  call  features.  Although 
there  were  intennittent  issues  with  VoIP,  the  participants  still  tended  to  use  the 
telephone  feature. 

•  The  participants  liked  the  ability  to  run  individual  exercises.  Due  to  their  OPTEMPO 
(Operational  Tempo)  and  the  difficulty  of  assembling  everyone  at  one  time,  the 
individual  training  feature  allows  students  to  train  as  their  schedule  allows. 
Participants  said  this  feature  would  create  a  better-trained  staff. 

•  The  participants  recommended  that  the  system  incorporate  a  feature  that  allows  them 
to  input  their  internal  reports  and  administrative  documents  easily  (e.g.,  SOPs, 
operation  orders  (OPORD),  message  formats,  and  briefing  formats). 

Eight  participants  completed  a  short  questionnaire  about  the  training  program  (see 
Appendix  D).  The  questionnaire  contained  Likert-type  scale  ratings  and  short  answers. 
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Participants  rated  their  perceived  improvement  in  performance  and  rated  items  pertaining  to  the 
training  exercise.  Short  answer  questions  pertained  to  expectations,  realism  of  training  content, 
and  usefulness  as  a  training  tool. 

Ratings  pertaining  to  participants’  perceived  improvement  in  their  perfonnance  averaged 
3.47  on  a  5 -point  Likert-type  scale  (1  =  none;  5  =  exceptional).  Participants  gave  the  lowest 
average  rating  (M=  3.25)  to  the  item  “Maintain  focus  on  mission  priorities.”  The  highest 
average  rating  (M=  3.75)  was  for  item  “Assess  the  situation  and  select  the  best  course(s)  of 
action.”  These  ratings  indicate  that  participants  saw  a  moderate  to  substantial  improvement  in 
performance. 

Participants  rated  comments  about  the  training  simulation  content  and  the  tool.  The  mean 
rating  was  4.13  on  a  5-point  Likert-type  scale  (1  =  strongly  disagree;  5  =  strongly  agree).  The 
item  with  the  lowest  average  rating  (M=  3.13)  was  “I  have  worked  harder  during  this  exercise 
than  in  most  exercises  I  have  participated  in.”  Although  this  average  was  in  the  “agree”  category 
it  is  still  low  considering  the  participants  were  engaging  in  an  exercise  created  for  higher-level 
JOC  commanders.  It  may  be  necessary  to  adjust  the  difficulty  level  of  these  and/or  future 
scenarios  to  match  the  skill  level  of  students.  The  highest  average  rating  (M=  4.75)  was  for  item 
“I  would  participant  in  training  using  the  simulation  in  the  future.”  This  provides  encouraging 
feedback  about  the  interest  future  participants  will  have  in  using  this  tool.  Participants  provided 
this  positive  rating  even  after  experiencing  technical  difficulties  with  the  tool.  Most  participants 
rated  the  simulation  environment  as  realistic  (M=  4.13),  and  all  participants  highly  recommend 
this  tool  be  incorporated  into  training/education  courses  (M=  5). 

Some  comments  to  improve  the  simulation  included: 

•  the  system  combined  all  elements  of  a  realistic  simulation  except  for  radio  traffic, 

•  the  system  should  provide  clearer  expectations  of  each  role  and  assets  available,  and 

•  it  is  a  good  system,  but  needs  longer  modules. 

Positive  feedback  included: 

•  the  training  system  helps  operators  crawl,  walk,  run; 

•  very  realistic  and  detailed  training; 

•  the  ability  to  tailor  training  to  specific  needs  and  roles  is  excellent;  and 

•  the  ease  in  which  the  users  can  modify  the  program  is  valuable  for  training  purposes. 

Global  Response  Conclusions 

The  TUF-D  is  a  collaborative  web-based  training  product  that  is  compatible  with 
government  computers  and  allows  participants  in  different  locations  to  train  and  communicate 
during  a  scenario.  The  product  includes  three  multi-agency  national  response  scenarios 
containing  challenging  and  realistic  events.  Along  with  the  scenarios,  the  system  provides  an 
instructional-overview  guide  that  describes  the  specific  behaviors  trained  within  the  scenarios 
and  provides  facilitator  guidance  to  increase  the  level  of  feedback  students  receive.  Included  in 
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the  instructional-overview  guide  is  a  user  guide  with  step-by-step  instruction  for  operating  and 
using  the  training  system. 

The  CTA  findings  provided  a  rich  data  set  that  allowed  for  the  development  of  an 
authentic  and  contextually  rich  simulated  training  environment  for  Anny  and  civilian  agencies 
who  must  collaboratively  respond  to  large-scale  crisis  events.  The  interface  replicates  the  real- 
world  decision  environments  of  responders  operating  in  JOCs  (or  EOC).  By  presenting  a 
desktop-style  decision  space,  responders  have  to  select  tasks  and  interpret  infonnation  as  they 
would  in  real  situations,  without  the  ability  to  view  the  scene  directly  and  with  infonnation 
filtered  through  on-scene  responders  and  the  media.  In  the  face  of  uncertainty  and  a  large 
amount  of  ambiguous  and  missing  infonnation,  it  is  vital  that  these  responders  be  able  to 
visualize  current  and  future  events  and  engage  in  adaptive  thinking  as  information  comes  in  and 
the  situation  changes.  . 

The  original  Red  Cape  themes  were  updated  based  on  the  results  of  the  present  research. 
While  the  context  in  the  updated  themes  match  the  original  themes,  the  new  theme  categories, 
and  the  identification  of  overarching  processes  provides  clarity  and  greater  distinction  between 
the  theme  definitions.  The  data  provided  behavioral  indicators  associated  with  each  theme. 

These  indicators  will  assist  training  developers  and  students  as  they  develop  training  and 
evaluate  perfonnance. 

Implementing  Unique  User  Requirements  and  Transitioning  Red  Cape  Training 

By  developing  a  rapid  modification  tool  for  the  web-based  collaborative  training 
platform,  scenario  developers  are  able  to  tailor  their  emergency  response  training.  Tailorable 
training  prepares  leaders  with  up-to-date  training,  provides  advanced  skills  in  novel  situations, 
and  shortens  the  time  from  skill  deficit  to  training  results.  The  research  resulted  in  a  training 
product  that  addresses  the  growing  number  of  critical  coordination  challenges  and  cultural 
differences  present  in  multi-agency  environments,  such  as  differences  in  mission  objectives, 
types  of  resources  available,  operational  vernacular,  SOPs,  areas  of  interest,  proximity,  and 
jurisdiction.  With  the  ability  to  develop  scenarios  targeted  toward  specific  training  goals, 
trainers  can  specify  learning  objectives  and  identify  the  outcomes  they  will  use  to  evaluate  skill 
development.  This  approach  provides  a  structured  method  to  address  the  challenges  surrounding 
multi-agency  responses  to  national  crisis  events. 

The  purpose  of  the  Implementing  Unique  User  Requirements  and  Transitioning  Red 
Cape  Training  research  was  to  develop  a  rapid,  web-based  scenario  development  and 
collaborative  training  tool  along  with  an  instructional  overview  guide  to  assist  scenario 
developers  in  understanding,  developing,  and  presenting  theme-based  training  to  support  both 
individual  and  collaborative  training.  As  with  the  Global  Response  research,  the  TUF-D 
software  architecture  was  utilized  to  create  an  authoring  tool  that  allows  users  to  create  or 
modify  scenarios  to  address  the  training  needs  in  the  operational  environment.  An  instructional- 
overview  guide  was  developed  to  explain  theme-based  training  and  provided  users  with  guidance 
on  how  to  create  using  the  authoring  tool.  By  focusing  on  specific  themes,  training  developers 
can  create  scenarios  that  target  specific  skills  and  adjust  conditions  to  suit  user  needs.  The 
development/modification  tool  was  developed  specifically  for  training  developers  and  training 
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administrators.  The  goal  of  the  tool  was  to  reduce  the  workload  of  training  developers  and 
administrators  when  creating  and  modifying  Red  Cape  vignettes  and  scenarios.  The  online  help 
system  walks  developers  and  administrators  through  the  vignette  creation  process  and  provides 
access  to  help  screens  and  instructional  aids.  Developers  and  administrators  can  access 
previously  created  scenarios  and  modify  them  to  fit  their  current  training  needs. 

Method 

Procedure 

The  TUF-D  did  not  contain  a  Vignette  Creation  Tool  prior  to  this  research.  An  authoring 
tool  for  was  developed  in  Adobe  Flex®.  The  tools  uses  a  thin  client  architecture — meaning  it 
does  not  require  software  installation  by  the  user.  The  Vignette  Creation  Tool  was  fully 
integrated  into  the  TUF-D  portal.  In  creating  the  tool,  the  manual  scenario  creation  process  used 
in  previous  research  was  adapted  into  a  self-contained,  step-by-step,  semi-automated  scenario 
development  process. 

The  design  and  development  of  the  instructional-overview  guide  was  based  on  systematic 
instructional  systems  design  (ISD)  principles  (Houser  &  DeLoach,  1998).  Content  was  designed 
to  promote  active  engagement,  encourage  self-reflection,  and  convey  the  personal  relevance  of 
knowledge.  Similar  to  the  TUF-D  interface,  the  instructional-overview  guide  interface  was 
designed  with  a  look  familiar  to  most  users.  The  interface  resembles  an  academic-style  textbook 
sitting  on  a  desk.  Each  page  incorporates  a  variety  of  text-based  instruction  and  interactive  flash- 
based  objects  to  accommodate  different  types  of  learners.  Text,  interactive  graphics,  and  videos 
accommodate  the  visual  learner  and  audio-based  graphics  for  the  auditory  learner. 

In  addition  to  considering  learner  types,  the  text-based  and  interactive  content  was 
designed  using  the  following  seven  principles  of  design  (Houser  &  DeLoach,  1998). 

1.  Contrast  is  the  difference  in  values,  colors,  textures,  shapes,  and  other  elements. 

Contrast  creates  visual  excitement,  increases  interest,  and  places  emphasis  on  content. 

2.  Emphasis  is  creating  a  center  of  interest  for  the  viewer.  The  center  of  interest  attracts 
attention  to  emphasize  its  importance  compared  to  the  other  elements  in  the  composition. 

3.  Balance  is  the  appearance  of  visual  equality  in  shape,  fonn,  value,  and  color.  Balance 
can  be  symmetrical,  asymmetrical,  or  radial. 

4.  Unity  aids  in  the  design  of  the  instruction  by  harmonizing  sections  and  providing 
content  cohesion. 

5.  Patterns  are  art  elements  that  use  planned  or  random  repetition  to  enhance 
composition  and  increase  users’  visual  experience. 

6.  Movement  is  the  visual  flow  of  the  content  by  object  placement  and  position 
throughout  composition. 

7.  Rhythm  is  the  repetition  of  visual  movement  in  terms  of  color,  shape,  and  lines. 

HyperText  Markup  Language  (HTML)  was  used  to  develop  the  instructional-overview 
guide  interface  and  Adobe  Flash®  was  used  to  create  customized  interactive  objects,  graphics, 
and  video.  Each  text-based  and  interactive  component  was  designed  using  the  six  elements  of 
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design:  color,  value,  texture,  shape,  form,  and  space  (Anderson,  1961).  The  six  design  elements 
compliment  the  seven  principles  of  design  used  to  create  the  text-based  and  interactive  content. 

Results  and  Discussion 


Vignette  Creation  Tool 

The  Vignette  Creation  Tool  allows  users  to  create  new  or  edit  existing  vignettes  (Figure 
4).  When  creating  new  vignettes,  users  specify  the  domain  and  roles  they  want  to  train  and 
create  timelines  for  their  vignettes.  Vignettes  are  typically  broken  down  into  one  or  more  time 
segments.  Time  progresses  during  segments  in  real  time.  Once  participants  have  addressed  all 
injects  within  a  segment,  they  can  fast  forward  in  time  to  the  next  segment.  When  creating 
scenarios,  users  place  injects  on  the  vignette  timeline  and  specify  the  parameters.  For  injects 
requiring  support  fdes  (e.g.,  html,  pdf,  or  video  fdes),  users  can  either  upload  fdes,  or  browse  a 
repository  of  reusable  injects  on  the  server. 


Figure  4.  Vignette  creation  tool. 

The  following  inject  types  are  available  through  the  Vignette  Creation  Tool. 

•  Important  Information:  This  html  page  displays  when  it  is  sent  from  the  simulation. 
This  page  typically  displays  an  introduction  before  the  start  of  the  vignette. 

•  TV  show:  Typically  a  short  video  news  story.  The  video  streams  to  users’ 
computers. 
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•  News  website  articles:  Displays  an  html  article  containing  text  and  images. 

•  Fax:  Displays  an  html  file  in  a  fax-type  format  containing  relevant  infonnation. 

•  Bulletin  board  postings:  Html  displays.  These  typically  contain  previous  warning 
orders  (WARNO),  Fragmentary  orders  (FRAGO),  commander  critical  infonnation 
requirements  (CCIR),  essential  elements  of  friendly  Information  (EEFI),  and  anything 
else  staff  might  post  on  the  walls  of  the  operations  center. 

•  Email  messages:  Incoming  Outlook-style  messages.  They  may  have  a  pdf 
attachment. 

•  Telephone  calls: 

free  response — the  user  can  type  in  any  response,  and 

question-based — the  user  is  asked  one  or  more  multiple  choice  questions.  The 
answers  are  part  of  the  assessment. 

•  Office  door  interaction:  Users  must  answer  knocks  at  the  door  to  see  an  image  and 
text.  Users  generally  use  this  feature  to  show  interaction  with  a  simulated  person  at 
the  scene.  The  simulated  entity  may  ask  multiple-choice  questions,  or  ask  for  input 
on  a  form,  such  as  a  press  release.  The  author  may  also  specify  a  free  response  text 
area. 

•  Meeting  interfaces. 

Simple  meeting  interface:  The  user  selects  whether  to  attend  the  meeting  or  not. 
The  meeting  pops  up  as  a  pdf  of  PowerPoint  slides. 

User  input  meeting  interface:  Depending  on  the  role  played,  a  simulated  or  real 
player  may  ask  users  to  fill  out  their  slide  for  an  upcoming  briefing  (e.g.,  Fill  out 
the  J1  slide  for  the  upcoming  Battle  Update  Briefing).  At  briefing  time,  the  user 
selects  whether  or  not  to  attend  the  meeting.  At  the  meeting,  the  system  presents 
the  set  of  slides  filled  out  by  live  and  simulated  players. 

The  Vignette  Creation  Tool  allows  authors  to  specify  the  tasks  they  want  to  evaluate. 
Authors  may  link  the  tasks  to  Red  Cape  themes.  They  can  do  this  in  the  assessment  section  of 
the  authoring  tool.  If  authors  assign  themes  to  each  task,  they  can  evaluate  perfonnance  based 
on  the  themes  in  the  AAR.  It  is  possible  to  add  new  modules  to  the  training  software  to  exercise 
different  military,  municipal,  state,  or  federal  organizations.  Team  training  modules  can  be 
executed  together  to  enable  collaborative  exercises  with  multiple  organizations.  Roles,  tasks, 
scenarios,  and  assessment  criteria  are  in  files  and  users  can  easily  alter  them  without  modifying 
the  source  code.  This  flexible  design  allows  for  easy  prototyping  and  modification  of  roles, 
scenarios,  and  questions.  Components  of  scenarios  include  pre-deployed  resources,  locations  for 
critical  events,  and  a  timeline  of  situations  that  contain  simulated  email  messages,  videos,  audio 
clips,  text  messages,  still  images,  and  other  informational  media. 

Instructional-Overview  Guide 

Organizations  frequently  do  not  understand  the  capability  gaps  that  exist  between  mission 
requirements  and  training  requirements  and,  although  they  may  often  train  crisis  action  response, 
they  do  not  include  the  skills  that  experienced  responders  use  to  resolve  crises  successfully.  The 
Instructional  Overview  was  created  to  include  information  about  the  importance  of  capturing  and 
accurately  portraying  expert  decision  processes  within  scenarios.  The  goal  of  the  guide  was  to 
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provide  trainers  with  an  introduction  about  how  to  implement  the  Red  Cape  Training  program 
and  how  this  method  leads  to  improved  performance. 

An  interesting  and  informative  instructional-overview  guide  is  essential  when  presenting 
the  capabilities,  strategies,  and  usefulness  of  a  training  package  to  users.  This  guide  contains 
video,  animations,  and  narrations  to  keep  the  audience  attentive  and  to  appeal  to  a  variety  of 
learning  styles.  The  instructional-overview  guide  incorporates  the  principles  of  immersion  and 
engagement  to  ensure  that  users  focus  on  the  foundational  content  pertaining  to  crisis  response 
and  interagency  collaboration.  The  guide  presents  the  theme-based  training  methodology  and 
provides  definitions  of  the  updated  themes  along  with  expanded  explanations  and  behavior 
indicators  of  each  theme.  The  instructional  overview  guide  also  provides  detailed  instruction  for 
operating  the  authoring  tool  and  running  exercises.  In  addition  to  the  instructional  overview 
guide,  the  authoring  tool  has  a  built-in  multimedia  help  system  that  provides  guided  instruction 
about  how  to  create  and  edit  vignettes  during  the  development  process. 

Scenario  Generation 

The  research  to  develop  a  scenario  generation  toll  achieved  two  purposes.  First,  an 
authoring  tool  was  created  to  rapidly  develop  and  modify  training  scenarios.  Second,  the  newly 
created  authoring  tool  was  utilized  to  create  the  scenarios  for  the  Global  Response  exercises. 
Using  the  authoring  tool  to  develop  our  own  exercises  allowed  the  research  team  to  test  and 
modify  the  tool  throughout  development. 

Scenario  developers  learned  to  use  the  tool  and  create  scenarios  during  teleconferences 
and  face-to-face  meetings.  During  meetings,  scenario  storylines  were  discussed  and 
development  issues  were  addressed.  After  several  meetings  the  scenario  development  team 
began  loading  injects  to  create  the  scenarios.  The  team  met  to  brainstonn  ideas  for  each  scenario 
based  on  incidents  described  by  participants  in  the  CTA  interviews.  The  team  developed  a  basic 
outline  of  each  vignette  with  a  focus  on  selected  emergency  response  themes.  Once  the  outlines 
were  prepared,  storyboards  were  produced  that  described  the  events  to  take  place  during  each 
scenario  along  with  the  injects,  dialogs,  and  multi -media  required  to  create  the  story.  The 
storyboards  were  tailored  to  fit  the  required  activities  for  each  role  in  the  TUF-D  scenario 
development  tool.  The  purpose  of  these  injects  was  to  create  complete,  multi-player  scenarios 
that  drive  students  to  take  actions  in  response  to  the  crises.  The  TUF-D  allowed  the  scenario 
development  team  to  input  injects  to  appropriate  staff  sections  and  ensure  that  each  scenario 
addressed  the  themes.  After  creating  the  scenarios,  they  were  tested  to  ensure  they  were 
complete,  accurate,  and  realistic.  The  review  process  occurred  several  times  before  products 
were  finalized.  During  the  iterative  process,  military  SMEs  reviewed  the  materials  and  made 
suggestions  and  recommendations  as  the  tool  evolved  to  the  final  product. 

Evaluation 

Method 

The  purpose  of  the  evaluation  was  to  test  the  Scenario  Authoring  Tool.  Evaluation 
participants  identified  issues  that  would  need  to  be  addressed  in  order  to  improve  the  application. 
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Participants.  The  same  participants  who  attended  Day  1  of  the  evaluation  and  evaluated 
the  Global  Response  training  also  attended  Day  2  to  evaluate  the  authoring  tool. 

Procedures.  Participants  attended  a  three-hour  session  on  Day  2  of  the  evaluation. 
Activities  for  Day  2  included  an  overview  and  instruction  on  using  the  scenario  developer, 
guided  practice  using  the  scenario  creation  tool,  participant  creation  of  scenarios  using  the 
scenario  creation  tool,  and  questionnaires. 

At  the  start  of  the  evaluation,  participants  sat  in  view  of  a  laptop  computer  and  received 
brief  instruction  on  the  scenario  development  features  of  the  training  system.  After  instructors 
provided  participants  with  a  quick  overview  of  their  experiences  developing  scenarios  for  the 
Global  Response  training,  participants  were  provided  a  “train-the-trainer”  session  on  how  to 
develop  a  scenario.  As  part  of  the  session,  the  participants,  with  facilitation  from  the  research 
team,  brainstonned  and  identified  actions  that  are  critical  to  perfonn  in  crisis  response  situations. 
As  participants  created  scenarios  using  the  tool,  the  research  team  provided  insight  on  Tactics, 
Techniques  and  Procedures  (TTP)  used  to  develop  the  Global  Response  scenarios.  During  the 
guided  practice  portion  of  the  evaluation,  participants  were  split  into  two  groups.  One  group 
brainstonned  ideas  and  generated  injects  for  one  timeline  segment.  The  second  group  created 
injects  by  using  the  information  created  by  the  first  group.  All  participants  had  hands-on 
experience  loading  injects  into  the  scenario  creation  tool. 

Results  and  Discussion 

The  participants  generally  had  no  problems  loading  injects  into  the  system.  They  learned 
quickly  how  to  use  the  various  features  to  create  injects.  Even  though  technical  problems 
occurred  while  participants  were  creating  scenarios,  they  frequently  commented  about  the  benefit 
of  the  authoring  tool.  Evidence  of  their  favorable  response  was  apparent  when  they  asked  the 
research  team  to  return  in  the  future  to  assist  them  in  loading  a  scenario  for  use  in  an  upcoming 
training  exercise. 

Three  participants  completed  a  short  questionnaire  about  the  scenario  creation  tool.  This 
questionnaire  contained  Likert-type  scale  ratings  and  short  answers  to  questions  (Appendix  E). 
The  low  number  of  respondents  can  be  attributed  to  delays  in  the  agenda  due  to  technical 
problems.  Participants  could  not  stay  late  to  complete  the  questionnaire.  The  data  for  the  three 
questionnaires  are  provided  with  the  understanding  that  the  averages  are  not  necessarily 
indicative  of  the  entire  sample. 

Ratings  pertaining  to  participants  perceived  improvement  in  their  ability  to  develop 
scenarios  averaged  4.13  on  a  5 -point  Likert- type  scale  (1  =  none;  5  =  exceptional).  Participants 
gave  the  lowest  average  rating  (M=  3.33)  to  the  item  “Conduct  an  assessment.”  The  highest 
average  rating  (M=  4.67)  was  for  items  “Identify  different  types  of  simulated  injects”  and  “Add, 
delete,  and  modify  simulation  injects.”  These  ratings  indicate  that  participants  understood  the 
notion  of  injects  and  how  to  apply  them  to  scenarios  in  a  useful  way. 
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Participants  rated  comments  about  the  scenario  development  tool.  The  mean  rating  was 
4.67  on  a  5-point  Likert-type  scale  (1  =  strongly  disagree;  5  =  strongly  agree).  The  item  with  the 
lowest  average  rating  (M=  4.33)  was  “The  scenario  developer  was  easy  to  use.”  This  relatively 
low  average  was  due  to  one  participant  rating  this  item  as  3  =  Neutral.  The  other  two 
participants  rated  this  item  as  5  =  Strongly  agree.  The  highest  average  rating  (M=  5)  was  for 
item  “I  would  use  the  scenario  developer  in  the  future.”  This  provides  encouraging  feedback 
about  the  interest  that  future  training  developers  will  have  in  using  this  tool  to  develop  scenarios. 
Participants  provided  this  positive  rating  even  after  experiencing  technical  difficulties  with  the 
tool.  With  only  three  respondents,  it  is  difficult  to  draw  specific  conclusions  or  to  determine  if 
the  entire  group  of  participants  had  the  same  general  opinion.  Clearly,  additional  data  should  be 
collected  before  any  conclusions  about  participant  opinions  of  the  scenario  developer  tool  can  be 
derived. 

Only  two  participants  provided  short  answers.  Some  responses  were  that  the  tool: 

•  is  user  friendly, 

•  has  group  play  interactivity, 

•  offers  quick  and  realistic  training, 

•  provides  a  good  way  to  build  scenarios  for  realistic  training,  and 

•  helped  to  gain  an  understanding  of  simulator  mechanics. 

Transitioning  the  Red  Cape  Project  Conclusions 

An  instructional  overview  guide  that  explains  the  purpose  and  benefits  of  theme-based 
training  and  provides  instruction  on  how  to  use  the  authoring  tool  was  developed.  The  guide 
uses  a  mix  of  multimedia  to  present  information.  The  overview  was  designed  using  a  modular 
approach  that  allows  direct  access  to  specific  sections  and  allows  users  to  progress  through  the 
guide  in  a  linear  or  a  non-linear  fashion. 

The  help  system  instructs  users  on  the  proper  use  of  theme-based  training  by  providing 
guided  help  throughout  vignette  creation.  The  guidance  encourages  users  to  build  authentic 
scenarios  that  embed  expert  themes  within  the  context  of  the  training.  The  online  help  system 
provides  clear  and  concise  instruction  on  how  to  use  the  various  features,  interfaces,  and  screens 
to  create  and  modify  vignettes.  The  system  does  not  limit  users  to  the  theme-based  approach;  it 
provides  users  with  guidance  to  create  a  variety  of  challenging  situations. 

The  result  is  a  synchronized  online  collaborative  training  program  that  supports  unity  of 
effort  for  project  development  and  unit  training  execution.  The  end  state  deliverable  trains 
leaders  to  understand  the  capabilities  of  Red  Cape  along  with  the  strategies  and  methods  for 
exploiting  them.  In  addition,  the  tool  allows  the  Army  and  the  National  Guard  to  tailor  scenarios 
to  address  current  and  changing  multi-agency  crisis  response  missions. 
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Garrison  Commander  Training 


As  part  of  the  Global  Response  research,  three  garrison  commander  in-class  exercises 
were  developed.  The  purpose  was  to  improve  current  in-class  critical  thinking  exercises  taught 
at  the  U.S.  Army  Management  Staff  College  by  implementing  a  theme-based  approach  to  teach 
crisis  response.  The  research  provided  an  opportunity  to  expand  our  understanding  of  theme- 
based  training.  The  objective  was  to  train  new  garrison  commanders  to  make  decisions  in 
ambiguous  situations  by  sorting  through  information,  prioritizing  tasks,  gathering  and  allocating 
resources,  coordinating  with  other  agencies,  and  predicting  outcomes.  Another  objective  was  to 
train  students  to  coordinate  and  synchronize  efforts  with  local  civil-military  and  interagency 
partners.  Three  emergency  response  situations  were  developed,  a  train  derailment,  a  bombing 
attack,  and  a  tornado.  The  scenarios  present  a  series  of  incoming  reports  about  the  event  and  the 
garrison  commander  students  must  detennine  how  to  manage  these  events.  At  various  points  in 
the  scenario  students  discuss  critical  issues  and  describe  the  courses  of  action  they  would  take 
based  on  the  information  presented. 

The  key  to  successful  learning  in  these  tasks  is  the  facilitation  of  questions  and 
discussions.  A  facilitator  guide  was  developed  that  provides  sets  of  initial  and  follow-up 
questions  in  a  manner  that  challenges  students  to  think  analytically  about  multiple  courses  of 
action  and  to  weigh  the  risks  and  benefits  of  decisions  while  considering  situational  factors  and 
unexpected  events.  The  structure  of  these  discussions  promote  thinking  skills  by  placing 
students  into  the  role  of  garrison  commander  and  requiring  them  to  respond  by  giving  guidance, 
making  requests,  and  deciding  on  courses  of  action.  The  exercises  focus  on  theme-based 
learning.  Refer  to  the  Red  Cape  Themes  Overview  (Appendix  C)  for  definitions  of  each  process 
and  theme.  Each  exercise  is  broken  down  into  segments  that  present  required  infonnation  to 
students.  Tasks  rarely  address  only  one  theme,  instead  they  form  the  complexities  present  in 
most  emergency  environments.  Many  themes,  such  as  “maintain  mission  priorities”  cross  over 
most  or  all  segments. 


Method 

Participants 

A  combination  of  input  of  government,  military,  and  research  team  personnel  participated 
in  the  development  of  the  scenarios  for  the  training  package.  Data  collected  during  the  29 
Global  Response  project  interviews  and  two  focus  groups  provided  the  data  used  to  create  the 
Garrison  Commander  vignettes.  In  addition,  observations  of  the  Garrison  Commander  Course, 
at  the  U.S.  Army  Management  Staff  College,  and  group  interviews  with  instructors  and  students 
were  conducted.  In  addition,  the  research  team  interviewed  an  additional  Garrison  Commander, 
one  Deputy  Garrison  Commander,  and  members  of  a  Garrison  and  Installation  Operation  Center 
(IOC)  staff  with  experience  in  emergency  response.  A  strong  collaborative  working  relationship 
was  established  with  the  interviewed  personnel,  providing  invaluable  insight  and  access  to  a 
garrison  crisis  response  team.  Observations  of  a  Garrison  Commander  and  staff  were  also 
conducted  while  they  were  undergoing  a  crisis  action  exercise. 
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Procedure 


Initially,  faculty  members  from  the  Garrison  Commanders  Course  at  the  U.S.  Army 
Management  Staff  College  were  interviewed  to  gain  a  better  understanding  of  the  research  need. 
Samples  of  the  materials  currently  used  in  the  Garrison  Commanders  Course  were  collected. 
Prior  to  this  research,  training  consisted  of  PowerPoint  slide  shows  with  simple  animation  and 
sound  effects.  The  goal  of  this  research  was  to  improve  the  training  received  by  garrison 
commanders  to  improve  their  ability  to  respond  to  crisis  response  situations  by  applying  the 
theme-based  method. 

To  create  the  garrison  commander  vignettes,  an  interface  architecture  that  enabled  the 
vignettes  to  run  over  the  Internet  or  on  a  self-contained  CD-Rom  was  developed.  The  goal  of  the 
garrison  commander  vignettes  was  to  present  content  that  promoted  reflection,  active 
engagement,  and  personal  relevance  of  knowledge.  To  immerse  students  in  the  content,  each 
vignette  introduction  provides  background  information  and  demographics.  The  vignettes  pause 
at  specific  points  so  instructors  can  ask  questions  that  engage  learners,  promote  reflection,  and 
engage  students  in  discussion.  The  interface  looks  like  an  interactive  PowerPoint  presentation 
with  forward  and  back  buttons  to  guide  users.  Each  page  incorporates  a  variety  of  text-based 
instruction  and  interactive  flash-based  objects  to  accommodate  different  types  of  learners.  As 
with  the  instructional  overview  guide  design,  the  seven  learning  principles  and  six  elements  of 
design  to  create  the  text-based  and  flash-based  interactive  content  were  utilized.  HTML, 
Extensible  Markup  Language  (XML)  was  used  to  develop  the  garrison  commander  vignettes  and 
Adobe  Plash®  to  create  customized  interactive  objects,  graphics,  and  video. 


Background  Information 


This  scenario  is  based  on  a  situation  that  unfolds  at  Fort  Knox,  Kentucky. 


Pigure  5.  Garrison  commander  vignette. 


Analysis 

Results  from  the  evaluation  of  the  current  training  and  a  behaviorally  science  review  of 
the  science  of  learning  allowed  the  research  team  to  identify  needed  improvements  to  the 
existing  training  garrison  commanders  received.  The  following  list  highlights  many  of  the 
identified  improvements. 

•  Gaps  in  skill/knowledge.  New  garrison  commanders  often: 

come  from  tactical  environments  and  have  not  been  in  command  of  garrison  units, 
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do  not  know  the  language  of  their  new  world, 
are  just  learning  city  management, 

do  not  know  where  to  go  during  an  event  -  to  the  scene  or  to  the  JOC, 

are  not  clear  about  what  to  do  in  the  first  hours  before  help  arrives  and  hours/days 

after  the  response  effort  is  underway  or  over, 

are  unclear  on  their  duties  vs.  the  incident  commander  vs.  the  mayor,  etc., 
do  not  have  contingency  plans, 

are  not  used  to  dealing  with  uncooperative  civilians,  and 

do  not  effectively  coordinate  with  the  senior  mission  commander. 

•  Garrison  commander  skill  requirements.  Garrison  commanders  need  to: 
develop  and  manage  relationships, 

work  outside  the  Garrison  with  other  agencies  and  officials, 
detennine  quickly  the  kinds  of  problems  they  need  to  solve, 
communicate  quickly  and  effectively  with  other  agencies, 
know  where  legitimate  authority  is,  and 

know  what  the  state  EOC  plan  is  so  they  do  not  duplicate  effort. 

Results  and  Discussion 

Comprehensive  storyboards  and  scenarios  for  three  crisis  action  vignettes  were 
developed.  The  storyboards  centered  on  training  goals  and  learning  objectives  identified  from 
the  interview  data.  Challenges  and  critical  decision  points  based  on  the  CTA  interview  data  were 
embedded  in  the  scenarios.  The  setting  for  the  three  garrison  commander  vignettes  was  Fort 
Knox,  KY.  The  Fort  Knox  Garrison  staff  and  the  Fort  Knox  IOC  provided  guidance  and  assisted 
the  research  team  in  gathering  images  for  the  scenarios.  Creating  scenarios  using  one 
operational  location  reduces  the  amount  of  background  information  students  must  keep  track  of 
as  they  work  through  multiple  scenarios.  The  background  information  is  the  foundation  upon 
which  new  information  is  considered.  To  immerse  students  in  a  situation,  students  receive 
frequent  Battle  Update  Briefs  (BUB),  or  staff  update  briefings,  to  present  infonnation  to  the 
garrison  commanders.  The  detailed  briefs  included  many  of  the  cues  students  need  to  visualize 
the  situations  presented  in  the  vignettes.  The  infonnation  provided  at  the  beginning  of  the 
vignettes  is  the  foundation  upon  which  students  consider  new  information. 

Scenario  Creation 

The  scenario  content  was  derived  from  incidents  described  in  the  CTA  interviews  with 
ganison  commanders  and  other  military/civil  emergency  responders.  Information  and  content 
from  U.S.  Anny  Management  Staff  College  products  and  from  consultation  with  the  Fort  Knox 
IOC  personnel  were  used  when  possible.  After  determining  the  crisis  response  skills  and 
associated  themes,  the  scenarios  were  developed  to  represent  the  required  skill  development. 

To  create  the  garrison  commander  vignettes,  the  team  met  in  person  to  brainstonn 
scenario  content.  Basic  outlines  for  each  scenario  were  created,  the  scenarios  on  storyboards 
were  scripted,  and  appropriate  media  content,  such  as  maps,  overlays,  and  visual  cues  were 
determined.  Various  members  of  the  team  reviewed  the  scenarios  and  revised  as  necessary. 
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After  completing  the  storyboards,  critical  thinking  questions  and  discussion  points  for  each 
section  of  the  scenario  were  created.  The  research  team  and  military  SMEs  reviewed  the 
scenarios  and  questions  and  provided  feedback.  The  review  process  went  through  several 
iterations  before  the  final  product  was  completed. 

Facilitator  Guide 

The  facilitator  guide  is  structured  the  same  for  each  of  the  three  exercises.  A  scene 
description  provides  a  high-level  account  of  the  scenario.  Following  the  description  is  a 
discussion  about  the  major  themes  addressed  in  the  exercises.  Each  segment  contains  critical 
thinking  questions  and  a  variety  of  discussion  points.  Facilitators  encourage  students  to  take 
notes  during  the  presentation  so  they  can  use  the  information  to  answer  questions  later.  Taking 
notes  teaches  students  to  listen  for  and  remember  useful  information  while  events  are  unfolding. 
The  guide  contains  a  large  number  of  discussion  points  that  address  a  variety  of  teaching  points. 
Facilitators  can  pick  from  the  discussion  points  or  create  their  own,  based  on  the  material  they 
want  to  highlight.  Figure  6  shows  an  exercise  segment,  accompanying  questions  and  discussion 
points. 


Actions? 

0 

Considerations 

®i 

IThe  time  is  1315  and  the  Garrison  Commander  (GC)  is  now  at  the  IOC 
coordinating  with  his  staff.  The  GC  requests  an  update  from  the  Battle  Captain  who  provides  details 
about  the  train  derailment  and  informs  him  that  all  members  of  the  IOC  are  present  except  the 
Director  of  Public  Works  (DPW)  who  is  stuck  on  the  other  side  of  the  derailment  site. 

Questions: 

1.  What  information  do  you  need  at  this  point?  What  questions  do  you  most  want  answered? 
Why? 

Discussion  points: 

Consider  gathering  information  regarding  the  train,  evacuations,  hazards,  personnel  on  scene 
Possible  questions  students  should  ask  include: 

Did  anyone  contact  the  railroad? 

What  chemicals  are  on  board  the  train? 

Do  we  have  a  manifest?  Does  it  list  the  hazardous  materials? 

How  can  I  obtain  the  manifest  or  other  information  about  the  hazardous  materials? 

Are  we  equipped  to  handle  a  chemical  spill?  What  resources  do  we  need? 

The  goal  of  this  question  is  to  increase  awareness  about  the  information  requirements  during 
emergency  incidents.  It  is  important  to  teach  students  what  information  to  ask  for,  how  to  ask  for 
that  information,  and  who  to  ask  for  each  piece  of  information. 

Themes  address:  Assess  -  collect  and  assess  information  and  understand  the  situation 


Figure  6.  Facilitator  Guide  questions  and  discussion  points. 


The  Facilitator  Guide  provides  a  brief  description  of  each  scenario  segment  followed  by 
segment  questions  and  discussion  points.  It  contains  information  about  how  to  leverage  the  Red 
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Cape  training  capabilities  and  expert  themes  to  deliver  knowledge  and  enhance  student 
experience.  It  provides  strategies  for  facilitating  exercises,  including  questioning  techniques, 
how  to  critique  decision-making,  and  how  to  promote  group  and  self-feedback.  The  Facilitator 
Guide  has  two  appendices,  one  provides  a  full  description  of  the  themes  and  the  other  is  a 
printable  version  of  the  segment  questions  and  discussion  points  with  room  for  facilitator  notes. 

Garrison  Commander  Training  Conclusions 

The  exemplar  training  created  for  the  U.S.  Anny  Management  Staff  College  includes 
three  scenarios  presented  using  multimedia  including  photos,  animation,  and  voice  narration. 
These  presentation  modes  accommodate  various  learning  styles  and  immerse  learners  in  the 
scenario.  Observing  the  current  theme-based  training  exercises  and  talking  to  instructors  and 
students  provided  insight  that  allowed  us  to  address  needs  specific  to  garrison  commanders.  An 
interactive  and  immersive  scenario  environment  was  developed  that  provided  facilitators  with 
targeted  questions  and  discussion  points  created  specifically  to  address  the  themes  and  enhance 
students’  critical  thinking. 

The  training  illustrates  the  diversity  of  theme-based  training.  In  addition  to  computer- 
based  training,  it  is  possible  to  incorporate  the  themes  into  facilitated  discussions  to  insure 
students  have  a  solid  understanding  of  the  themes  and  their  associated  cognitive  skills.  Previous 
Red  Cape  training  within  the  Garrison  Commander  Course  introduced  the  themes,  and  this 
update  extends  that  training  to  address  learning  styles,  incorporate  multi-media,  and  provide  an 
immersive  learning  environment. 


General  Discussion 

This  research  and  development  project  extended  the  original  theme-based  training 
developed  by  ARI  to  include  computer-based  simulation  training,  a  computerized  scenario 
creation  tool,  and  multi-media  in-class  training  scenarios  for  garrison  commanders.  The  research 
results  illustrate  the  flexible  nature  of  theme-based  training.  The  computer-based  training 
leveraged  a  pre-existing  National  Guard  training  product  to  present  individuals  and  groups 
complex  scenarios  that  incorporate  the  themes  into  situations.  As  students  train  to  the  themes, 
their  actions  indicate  their  theme-based  behaviors,  providing  insight  into  the  cognitive  skills  in 
which  they  are  proficient  and  the  skills  in  which  they  need  further  training.  The  focus  of  the 
Global  Response  training  is  on  the  behaviors  displayed  by  students  in  response  to  situational 
factors.  They  increase  their  theme-based  skills  by  taking  action;  thus,  they  learn  from 
experience.  By  first  using  CTA  to  collect  and  analyze  the  critical  decisions  and  tasks  of 
emergency  responders,  scenarios  were  created  that  replicate  real-world  crises,  including  the 
tough  challenges,  unexpected  consequences,  and  the  strategies  used  by  experienced  responders. 

The  Global  Response  training  provides  instructors  with  ready-made  scenarios.  This  is  a 
good  way  to  introduce  responders  to  the  tool  and  allows  for  instant  access  to  training.  However, 
as  their  familiarity  with  the  tool  increases  and  training  needs  change,  the  exercises  would 
become  obsolete.  To  counter  this,  instructors  were  provided  with  the  capability  to  modify  these 
scenarios  or  create  their  own  scenarios.  Thus,  this  tool  will  remain  relevant  as  training  needs 
evolve  to  accommodate  the  changing  needs.  The  Vignette  Creation  Tool  is  a  user-friendly  tool 
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that  allows  trainers  to  create  and  customize  exercise  vignettes  to  fit  current  and  evolving  training 
needs.  With  this  tool,  trainers  are  able  to  link  scenario  injects  and  expected  actions  to  behavioral 
themes.  The  scenario  creation  tool  is  an  important  enhancement  to  the  simulation-based  training 
program  and  contributes  to  command-level  response  training  beyond  this  research — allowing  the 
National  Guard  to  develop  exercises  for  a  variety  of  requirements. 

Applying  theme-based  training  to  classroom  settings  allows  for  facilitated  group 
discussions  that  enhance  critical  thinking  skills  and  provide  a  deeper  understanding  of  the 
themes.  Three  exercises  for  garrison  commander  training  were  created  to  advance  their  current 
Red  Cape  exercises.  Facilitators  were  provided  with  critical  thinking  questions  and  discussion 
points  after  each  exercise  segment.  These  questions  stem  directly  from  the  CTA  interview  data 
and  challenges  students  to  think  about  a  variety  of  situational  factors,  weigh  multiple  strategies, 
and  contemplate  various  courses  of  action.  A  comprehensive  facilitator  guide  explains  the 
themes  and  identifies  the  major  themes  addressed  in  each  exercise  segment. 

As  a  package,  these  three  training  products  provide  a  comprehensive  training  program  in 
multi-agency  emergency  response.  They  each  engage  students  in  deliberate  practice  by  allowing 
for  repeated  perfonnance  and  constant  immediate  feedback  about  performance  on  difficult 
aspects  of  emergency  response.  Trainers  are  able  to  structure  scenarios  to  address  students’ 
needs  and  can  inject  questions  to  facilitate  performance  feedback  during  both  the  computer- 
based  and  in-class  exercises. 

Participant  response  was  positive,  though  more  evaluation  is  necessary  to  fully  assess 
usability  and  ascertain  if  gaps  in  training  still  exist.  The  following  bulleted  list  provides  other 
limitations  identified  during  the  research. 

•  Scenario  developers  currently  do  not  have  access  to  news  reports  and  other  multi- 
media  similar  to  those  present  in  the  scenarios.  It  would  be  useful  to  develop  a 
repository  of  multi-media  content  where  developers  can  store  and  find  multi-media  to 
fit  their  needs. 

•  The  focus  of  the  vignette  creation  tool  was  on  scenario  development.  The  scope  of 
this  effort  did  not  allow  for  the  creation  of  a  tool  that  assists  with  scenario  design.  A 
design  tool  would  walk  users  through  the  design  process,  including  picking  a  scenario 
topic  (i.e.,  earthquake),  choosing  roles,  and  thinking  up  creative  injects  and 
consequences  that  address  specific  themes.  The  instructional  overview  guide 
provides  some  guidance,  but  not  to  the  extent  as  would  an  automated  tool. 

•  The  VoIP  is  a  useful  addition  to  this  tool;  however,  it  is  currently  not  possible  to 
record  students’  conversations.  Without  the  ability  to  track  what  students  say  to  each 
other,  instructors  and  students  cannot  evaluate  these  conversations  and  provide 
feedback.  If  a  method  to  gather  this  infonnation  becomes  available  in  the  future,  it 
would  be  beneficial  to  add  it  to  the  tool.  With  this  information,  instructors  and 
students  can  evaluate  question  phrasing,  tone  of  voice,  command  style,  and  use  of 
themes. 
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These  limitations  suggest  future  work  stemming  from  this  research.  In  addition  to  the 
potential  extensions  to  these  tools  mentioned  above,  the  following  bulleted  list  provides  other 
future  work  recommendations. 

•  Further  enhancement  of  the  AAR  to  include  a  more  detailed  analysis  of  actions, 
including  e-mail  and  telephone  call  content,  explicit  explanation  of  the  pros  and  cons 
to  each  action  choice  (both  the  actions  taken  and  those  not  taken),  and  automated 
annotation  in  addition  to  instructor  annotation. 

•  A  web-based  portal  for  trainers  to  share  the  vignettes  they  create,  multi-media,  ideas 
for  training,  questions  about  training  issues,  etc. 

•  Extend  the  in-class  garrison  commander  training  to  a  computer-based  exercise 
program.  Conducting  in-class  exercises  followed  by  computer-based  training  insures 
that  student  skill  level  matches  the  training  level.  Similarly,  creating  comprehensive 
in-class  exercises  for  military  and  other  emergency  responders  will  introduce  the 
skills  needed  to  perform  in  the  computer-based  exercises  and  beyond. 

•  Evaluation  participants  made  suggestions  that  can  be  easily  incorporated  into  the  tool 
in  the  future.  Participants  would  like  the  capability  to  use  radios  within  the  exercises. 
They  would  also  like  better  definitions  of  each  role  within  the  scenario  and  greater 
clarity  on  the  assets  available  throughout  the  scenario. 

This  training  package  advances  the  theme-based  method,  provides  much  needed  training 
for  emergency  responders  from  multiple  agencies  whose  job  it  is  to  resolve  the  nation’s  biggest 
crises.  Thankfully,  these  events  are  not  common,  however,  when  such  events  do  occur,  our 
national  and  military  responders  must  be  prepared.  The  best  preparation  comes  from  experience. 
The  exercises  created  with  these  training  tools  provide  that  experience-based  training.  This 
allows  responders  to  develop  the  mental  models  necessary  to  become  proficient  prior  to  actual 
events.  By  providing  consistent  theme-based  training  of  complex  cognitive  skills  and  behaviors, 
we  reduce  “on-the-job-training”  for  critical  job  requirements. 
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Appendix  A 


Think  Under  Fire  (TUF)  Decisions 

A  web-based  simulation  system  for  exercising  critical  decision-making  skills. 
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Description.  The  TUF  Decisions  is  a  simulation- 
based  training  system  that  allows  individuals  or 
teams  to  make  decisions  in  realistic  situations  and 
see  the  outcomes.  The  TUF  Decisions  is 
completely  web-based  with  no  software  install;  the 
user  needs  only  a  broadband  connection  and  java- 
enabled  browser  such  as  Internet  Explorer  to  join 
an  exercise.  Each  exercise  scenario  presents  the 
unfolding  situation  through  realistic  injects 
including  email,  news  videos,  website  articles,  and 
simulated  telephone  calls.  Users  make  decisions 
and  mitigate  the  incident  by  responding  to 
messages,  communicating  with  teammates,  and 
giving  commands  to  simulated  entities. 

Uses.  The  TUF  Decisions  is  data  driven,  and  can 
exercise  any  domain.  The  TUF  Decisions  have  been  used  to  exercise  National  Guard  Joint  Operations 
Center  staff,  Public  Elealth  and  Emergency  Management  personnel,  and 
Transportation  professionals,  among  others.  Injects  can  be  written  with 
specific  answers  for  knowledge  acquisition  and  testing,  or  free  form 
responses  for  exercise  play. 

Running  an  exercise.  The  TUF  Decisions  exercises  are  accessed  through 
a  web  portal.  The  user  logs  in  to  run  an  individual  exercise  scenario,  join  a 
scheduled  team  exercise,  take  a  tutorial,  or  access  training  records.  For  an 
individual  exercise,  the  user  selects  a  scenario,  a  role  to  play,  and  a  state 
and  county  to  exercise  in.  If  the  user  selects  a  “novice”  skill  level,  TUF 
Decisions  will  display  task  prompts  that  lead  the  user  through  the  exercise. 

Learning  approach.  Research  reveals  that  experience  plays  a  central  role 
in  making  effective  decisions,  especially  during  critical  situations.  The 
TUF  Decisions  exercises  build  a  pattern  base  that  allows  decision  makers 
to  make  better  recognitional  decisions  when  real  world  challenges  arise. 

The  TUF  Decisions  offers  progressive  training,  from  acquisition  of 
knowledge  through  informational  presentations,  to  building  of  skills  in  an 
individual  role  trainer  with  simulated  teammates,  to  formulating  abilities 
by  exercising  with  a  team  in  a  real-world  environment. 

Concrete  feedback  is  required  to  build  expertise.  Each  role  is  assessed  on 
taking  the  steps  to  complete  cognitive  tasks  such  as  maintaining  control  of  the  scene,  communicating 
both  within  the  team  and  with 
outside  agencies,  and  utilizing 
resources.  Responsibilities  may 
overlap  between  team  members; 
the  assessment  reveals  how  team 
members  work  together  to  achieve 
overall  goals. 


Assign  creation  ofTask  Assignment  Forms  for  received  missions  and  distribute 
the  completed  TAF  to  assigned  units. 

Note:  Time  ran  out  to  do  this  task. 


Update  the  BUB  J3  and  Force  Protection  slides 
Completed  By:  Andy 


Get  a  state  tracking  number  for  every  mission. 
Note:  A  reminder  was  sent. 

To  Be  Completed  By:  JFHQ.BC 
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Global  Response:  Large  Scale  Interagency  Training 
Interview  Guide:  Camp  Atterbury 

Interview  Objectives 

Objective  1 :  Understand  the  difficulties  National  Guard  responders  at  the  operational  level  face 
when  making  decisions  in  multi-agency  emergency  response  environments 

•  Identify  critical  tasks. 

•  Identify  challenges  to  interagency  coordination  and  response. 

•  Identify  the  cues  and  factors  that  contribute  to  effective  response  during  crisis  events. 

Objective  2:  Clarify  current  training  needs  and  gaps  in  training  for  military  response  to  multi¬ 
agency  crisis  events 

•  Identify  areas  where  decision  skills  can  be  enhanced. 

•  Identify  training  scenarios  that  would  add  benefit  to  crisis  action  training. 

•  Identify  training  challenges. 


Probe  Questions 

Ask  for  the  most  difficult  challenge  first.  If  time,  ask  for  the  next  difficult 
challenge  and  so  on.  If  they  say  there  were  no  other  difficulties,  then  ask  for  the 
first  decision  point  involving  crisis  action  response,  the  second  decision  point,  etc. 


Part  I 

Briefly  describe  a  multi-agency  emergency  response  situation  where  you  were  particularly 
challenged  to  make  difficult  decisions. 

What  was  the  most  difficult  challenge  (challenging  decision)  you  faced  in  this  situation  ? 

•  What  made  this  challenge  difficult? 

•  What  was  your  main  objective  during  this  task? 

•  How  did  this  challenge  impact  your  ability  to  achieve  this  task? 

•  What  was  your  biggest  concern  when  approaching  this  task? 

•  How  did  working  with  other  agencies  influence  your  ability  to  act  (positive/negative)? 

Which  cues/factors  did  you  consider  as  you  attempted  to  complete  this  task? 

•  How  did  you  decide  to  approach  this  problem? 

•  What  specific  factors  played  into  this  decision? 

•  What  did  you  see  that  led  you  to  choose  this  course  of  action  (CO  A)? 

•  What  information  did  you  use  in  making  this  decision? 

What  is  the  first  action  you  took  when  confronted  with  this  challenge? 

•  What  did  you  expect  to  happen  when  you  took  this  action? 
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•  Why  do  you  think  things  happened  the  way  you  did/did  not  intend? 

•  What  other  COAs  did  you  consider?  Why  did  you  rule  them  out? 

•  Are  there  any  alternative  actions  that  might  have  worked? 

How  might  an  inexperienced  responder  (or  someone  with  no  experience)  handle  this  task? 

•  What  errors  might  an  inexperienced  person  make  in  your  position  in  this  situation 

•  What  factors/cues/indicators  would  someone  with  little  experience  miss  as  he/she  tried  to 
accomplish  this  task? 

•  What  do  you  know  now  that  you  wish  you  knew  when  you  started  as  an  operations 
officer? 

Repeat  questions  for  the  next  challenging  task,  and  the  next,  etc.,  as  time  permits 
Part  II:  Instructional  Challenges 

What  are  the  biggest  challenges  faced  when  preparing  responders  for  real-life  large-scale 
incidents? 

•  Describe  any  decision-making  challenges  that  are  particularly  tough  for  new 
responders. 

•  Describe  any  challenges  faced  by  new  responders  when  they  work  with  multiple 
agencies  during  emergencies. 

•  What  are  the  most  difficult  tasks  to  accomplish  when  training  for  large-scale 
response? 

•  Are  there  any  training  areas  where  you  would  like  to  see  more/different  training? 

o  Describe  the  training  components  you  think  are  the  most  useful, 
o  Describe  any  training  components/scenarios  you  would  like  to  see  added  in 
the  future. 

If  you  were  going  to  create  a  training  scenario  that  addresses  some  of  the  challenges  described 
today,  what  would  you  put  in  that  scenario? 

•  What  factors  would  add  ambiguity  or  uncertainty  to  this  situation? 

•  To  reach  a  successful  outcome,  what  are  the  critical  tasks  that  need  to  be  completed? 

•  What  cues  and  factors  should  trainees  focus  on  to  understand  and  make  decisions 
about  this  scenario? 

•  What  vital  pieces  of  information  should  they  seek  out  as  they  deal  with  the  situation? 

•  What  are  the  possible  courses  of  action  they  can  take  that  will  lead  to  a  satisfactory 
outcome? 


B-2 


Appendix  C 


Red  Cape  Themes  Overview 

Theme-based  training  targets  specific  skills  by  identifying  the  cognitive  skills  of  experienced 
decision  makers  and  categorizing  them  into  themes.  Themes  form  the  thinking  skills  required 
within  a  given  set  of  circumstances.  Theme-based  training  provides  instructors  with  a  method 
for  training  complex  critical  thinking  skills.  By  training  to  themes,  students  increase  ability  to 
assess  situations,  synchronize  personnel,  assets,  and  agencies,  and  execute  plans  in  a  timely 
manner.  The  Red  Cape  themes  presented  below  describe  the  processes  and  cognitive  activities 
required  to  perform  in  multi-agency  emergency  response  environments.  Experienced  National 
Guard,  Federal,  and  local  emergency  responders  provided  infonnation  about  their  decision 
experiences  in  actual  events,  such  as  Hurricane  Katrina,  tornadoes,  floods,  airplane  crashes,  and 
wild  fires.  This  information  was  synthesized  to  form  the  Red  Cape  themes  presented  below. 
The  eight  themes  are  categorized  into  three  processes  key  to  emergency  response:  Assess, 
synchronize,  and  execute. 

Processes:  A  series  of  interdependent  actions  executed  during  multi-agency  response  to  crisis 
events.  These  actions  form  an  evolving  response  pattern  aimed  at  resolving  the  crisis.  Through 
these  actions,  it  is  possible  to  employ,  maintain,  and  revise  plans.  These  processes  involve 
activities  such  as  leveraging  previously  established  relationships,  and  establishing  morale  and  a 
battle  rhythm  that  promotes  effective  inter-  and  multi-agency  response.  There  are  three 
processes:  Assess,  Synchronize,  and  Execute.  Themes  are  associated  with  each  process. 

Themes:  The  expert  cognitive  activities  that  responders  engage  in  to  achieve  successful 
resolution  of  the  crisis.  These  themes  illustrate  the  processes  and  provide  behavioral  patterns 
through  which  responders  can  model  their  understanding,  plans,  and  decisions. 

Definition  of  Processes 

1.  Assess:  To  analyze  and  process  relevant  information  in  a  crisis  event  in  order  to  develop 
situational  understanding  and  take  action  appropriate  to  the  current  and  unfolding  situation. 

Themes 

1 .  Collect  and  assess  information  and  understand  the  situation. 

2.  See  the  big  picture  and  predict  events. 

2.  Synchronize:  To  bring  together  necessary  assets,  personnel,  infonnation,  and  action  steps  in 
order  to  create  a  team  and  environment  that  promotes  effective  planning,  communication,  and 
response  to  the  situation. 

Themes 

1.  Coordinate  internally  and  externally  across  all  agencies. 

2.  Acquire,  prioritize  and  allocate  assets. 

3.  Establish  chain-of-command. 
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3.  Execute:  To  take  action  at  appropriate  times  and  in  appropriate  manners  in  order  to  affect 
change  that  drives  the  situation  to  a  satisfactory  conclusion. 

Themes 

1 .  Recognize  decision  points. 

2.  Address  public  and  responder  needs. 

3.  Maintain  mission  priorities. 

Themes  Associated  with  Each  Process 

Assess 

1.  Collect  and  assess  information  and  understand  the  situation. 

Definition:  Effective  commanders  and  staff  are  able  to  quickly  detennine  how  and  where  to 
collect  the  information  necessary  to  develop  a  situational  understanding.  They  are  able  to 
quickly  evaluate  and  prioritize  incoming  infonnation  and  construct  a  coherent  picture  of 
unfolding  events.  They  draw  on  their  experiences,  along  with  the  experiences  of  those  around 
them,  to  understand  the  situation.  They  focus  on  important  cues  to  interpret  the  current  situation 
and  formulate  plans  that  leverage  these  situational  elements  in  order  to  resolve  the  crisis. 

Explanation:  Decision  makers  must  make  sense  of  what  is  going  on  in  a  situation  before  they 
can  make  decisions  and  take  action.  They  do  this  by  scanning  the  environment  for  clues  that  will 
help  them  understand  what  is  occurring.  They  look  for  patterns  that  they  recognize  and  then 
construct  a  story  that  fits  their  existing  knowledge.  Patterns  consist  of  all  the  things  present  in 
the  situation  including  sights,  sounds,  smells,  human  behaviors,  environmental  conditions,  and 
incoming  reports.  The  amount  of  infonnation  present  in  critical  events  is  more  than  any 
individual  person  can  perceive  and  process.  They  must  choose  which  pieces  of  infonnation  they 
attend  to  and  what  to  filter  out.  The  ability  to  gather  and  sort  information  develops  with  practice 
and  experience. 

Behavioral  indicators:  Someone  with  a  good  understanding  of  the  situation  who  is  adequately 
filtering  information  will  display  indicators  such  as: 

•  Issue  orders  or  offer  guidance  to  deal  with  future  events,  not  just  immediate  concerns. 

•  No  longer  seek  information  in  order  to  define  the  situation. 

•  Actively  try  to  understand  the  progress  and  consequences  of  earlier  decisions. 

•  Ask  direct,  specific  questions  rather  than  general  questions  (i.e.,  “What  is  going  on?”). 

•  Focus  on  specific  incoming  infonnation  and  filter  out  less  important  information. 

Instructor  example:  To  train  students  to  collect  and  assess  information  and  understand  the 
situation,  the  instructor  can  insert  a  report  into  the  scenario  about  an  incident,  such  as  a 
suspicious  vehicle  sighting.  The  student  would  then  need  to  request  information  from  others, 
assess  that  information,  and  take  action  appropriate  to  the  infonnation  they  received.  The 
instructor  can  track  these  activities  during  the  exercise  and  in  the  after  action  report. 
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2.  See  the  big  picture  and  predict  events. 

Definition:  Effective  commanders  and  staff  construct  a  coherent  picture  of  unfolding  events  and 
see  the  overarching  implications  of  possible  actions.  They  take  a  global  view  of  the  situation  and 
predict  the  consequences  associated  with  various  responses  or  non-responses. 

Explanation:  When  a  decision  maker  only  focuses  on  the  immediate  problem  without  looking  at 
the  problem  in  the  context  of  the  entire  situation,  their  decisions  will  likely  have  unintended 
consequences.  Experts  tend  to  see  more  than  just  bits  and  pieces  of  the  problem  by  seeing  the 
entire  picture  in  one  complete  snapshot  and  creating  a  story  around  it.  The  story  tells  them  what 
has  gone  on  to  create  the  current  situation  and  what  they  must  do  to  achieve  a  successful 
outcome.  This  story  also  allows  them  to  evaluate  action  choices  and  predict  the  positive  and 
negative  outcomes.  Those  who  do  not  make  this  “holistic”  evaluation  are  likely  to  see  only 
pieces  of  the  picture  and  fail  to  anticipate  what  will  happen  next. 

Behavioral  indicators:  You  can  expect  someone  who  is  seeing  the  big  picture  to: 

•  Focus  on  larger  concerns,  rather  than  deliberating  over  only  small  matters. 

•  Take  into  consideration  the  consequences  of  their  actions  to  outside  agencies  and  the 
community. 

•  Make  interpretations  beyond  observable  facts. 

•  Create  a  story  from  small  bits  of  infonnation. 

•  Not  focus  on  only  one  aspect  of  the  situation  (i.e.,  finances,  communication 
equipment,  traffic  management). 

•  Take  effective  action  when  new  infonnation  comes  in. 

•  Set  a  clear  battle  rhythm  and  keep  motivation  up. 

Instructor  example:  To  train  students  to  see  the  big  picture  and  predict  events,  the  instructor  can 
create  an  e-mail  that  infonns  students  that  toxic  chemicals  are  leaking  into  the  Post’s  water 
supply.  The  main  response  may  be  to  shut  off  the  water  supply,  but  the  big  picture  might  show 
that  turning  off  the  water  supply  will  do  more  damage  in  terms  of  health  risks,  costs,  and 
infrastructure  dysfunction  (i.e.,  patient  care  at  the  hospital).  If  the  student  chooses  to  turn  the 
water  off  without  mitigating  these  risks,  the  instructor  should  create  injects,  such  as  e-mails,  with 
reports  of  these  consequences. 

Synchronize 

1.  Coordinate  and  communicate  internally  and  externally  across  all  agencies. 

Definition:  Effective  commanders  and  staff  collaborate  within  and  across  agencies  in  a  manner 
that  develops  a  shared  understanding  of  the  mission  and  of  responder  roles,  while  enabling  the 
synchronization  of  mission  goals  and  objectives.  They  use  relationships  and  diplomacy  to 
identify  and  minimize  collaborative  barriers  and  to  produce  maximum  synergy.  They 
communicate  in  a  timely  matter,  use  a  common  language  across  agencies,  and  find  workable 
solutions  to  communication  barriers  such  as  lack  of  interoperability. 
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Explanation:  It  is  necessary  to  coordinate  and  communicate  with  all  responding  agencies. 
Collaboration  means  more  than  just  working  side-by-side  with  each  other,  it  means 
understanding  what  others  need  to  operate  effectively.  This  promotes  a  shared  understanding  of 
the  task  and  responder  roles.  It  is  important  to  leverage  established  relationships  among 
community  leaders  in  order  to  establish  community  support  and  cooperation.  It  is  important 
what  decision  makers  say  and  how  they  say  it.  Effective  decision  makers  use  diplomacy  to 
convey  an  understanding  of  the  public  need  even  while  they  are  requesting  assistance  and  taking 
control  of  turbulent  situations.  When  decision  makers  convey  information  also  affects 
operations.  If  decision  makers  withhold  infonnation  for  too  long,  erroneous  and  detrimental 
actions  may  result.  If  they  give  out  information  too  soon,  it  may  not  be  factual,  or  it  may  be 
useless  without  additional  information. 

Behavioral  indicators:  You  can  expect  someone  who  is  coordinating  and  communicating  across 
agencies  to: 

•  Speak  plain  English,  avoiding  acronyms  and  agency-specific  language. 

•  Communicate  with  outside  agencies  and  political  leaders  in  the  community. 

•  Establish  a  shared  communication  system. 

•  Adapt  to  communication  equipment  failures  (i.e.,  sending  a  runner,  word-of-mouth 
spread  of  information). 

•  Adapt  communication  style  to  fit  audience  (i.e.,  political  leaders  vs.  staff). 

•  Hold  regular  briefings  and  huddles  to  utilize  the  experiences  of  others  in  the  room. 

•  Calls  upon  the  relationships  fostered  prior  to  the  event. 

•  Make  clear  the  chain  of  command,  who  people  should  report  to,  where  people  should 
report. 

Instructor  example:  To  teach  students  to  coordinate  internally  and  externally  across  all 
agencies,  the  instructor  might  create  an  inject  where  a  staff  member  knocks  at  the  student’s  door 
and  states  that  the  incident  commander  (Fire  Chief)  is  not  allowing  emergency  responders  from 
the  local  community  on  base  to  help  fight  fires,  care  for  the  wounded,  and  keep  the  Post  secure. 
The  Fire  Chief  did  this  even  though  the  Major  General  requested  outside  responders’  assistance. 
The  student  should  make  calls  and  send  e-mails  to  find  out  why  this  is  occurring  and  work  to 
clear  up  any  misunderstandings,  coordinate  requests,  and  resolve  the  issue  in  a  manner  that 
creates  cooperation  between  agencies. 

2.  Acquire,  prioritize,  and  allocate  assets. 

Definition:  Effective  commanders  and  staff  determine  the  type  and  amount  of  equipment  and 
personnel  that  is  necessary  to  accomplish  the  mission.  They  are  effective  in  acquiring, 
prioritizing  and  deploying  the  necessary  assets.  They  think  creatively  to  use  assets  that  are 
readily  available  and  take  advantage  of  the  expertise  around  them  in  order  to  mitigate  the 
situation. 

Explanation:  Thinking  creatively  during  a  crisis  allows  decision  makers  the  flexibility  to  find 
solutions  when  assets  are  not  available  as  planned.  Thinking  creatively  involves  acquiring  assets 
from  atypical  sources  and  using  available  assets  for  unconventional  purposes.  They  are  aware  of 
what  assets  are  available  and  can  predict  when  resources  will  run  out.  They  are  able  to  allocate 
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assets  to  the  appropriate  locations  at  the  appropriate  times  and  in  the  appropriate  quantities. 
Assets  include  human  assets,  and  effective  decision  makers  utilize  the  unique  skills  of  those 
around  them.  This  is  especially  important  when  responders  have  careers  outside  of  their 
responder  role. 

Behavioral  indicators:  You  can  expect  someone  who  is  effectively  managing  assets  to: 

•  Maintain  a  realistic  assessment  of  asset  availability. 

•  Be  aware  of  limits  to  resources. 

•  Explore  options  for  obtaining  assets. 

•  Prioritize  asset  distribution  to  meet  critical  and  long-tenn  needs. 

•  Not  just  “throw  assets  at  the  problem.” 

•  Knows  when  to  call  in  support  from  external  agencies. 

•  Get  advice  from  others  and  use  their  experiences  as  a  guide. 

•  Appropriately  ration  resources,  including  human  resources. 

Instructor  example:  To  teach  students  to  acquire,  prioritize,  and  allocate  assets,  the  instructor 
can  create  a  scenario  that  puts  a  severe  strain  on  resources,  such  as  multiple  explosions  at 
different  locations  around  Post.  Students  should  request  regular  updates  on  the  status  of 
available  resources  and  actively  look  to  acquire  resources  from  other  agencies. 

3.  Establish  chain-of-command. 

Definition:  Effective  commanders  and  staff  make  clear  to  all  personnel  who  is  in  charge  of  an 
incident.  They  remain  flexible,  build  strong  teams,  and  create  an  environment  that  allows 
subordinates  leeway  to  share  information  and  ideas,  make  decisions,  and  complete  tasks.  They 
clearly  and  concisely  express  intent  and  mission  requirements,  giving  orders  and  direction  that 
are  specific  enough  to  be  clear,  but  flexible  enough  for  personnel  to  achieve  under  dynamic 
situational  constraints.  They  are  sensitive  to  political  and  community  relationships  and  to  the 
requirements  of  higher  commanders. 

Explanation:  A  clear  understanding  of  who  is  in  charge  establishes  needed  structure  and  clarity 
during  a  crisis.  Decision  makers  need  to  make  clear  what  are  each  responder’s  responsibilities. 

It  is  important  that  decision  makers  remain  flexible  as  responders  handle  their  responsibilities  in 
their  own  manner  and  that  responders  know  they  have  leeway  to  think  creatively  and  proactively 
in  changing  situations.  All  agencies  should  have  a  shared  understanding  of  the  mission 
requirements.  Decision  makers  need  to  express  empathy  and  patience  toward  their  staff 
members  and  members  of  other  agencies,  particularly  when  the  emergency  is  local.  Unlike  in 
overseas  operations,  local  events  have  direct  impact  on  responders’  families,  friends,  and 
property.  Even  though  a  decision  maker  may  be  in  charge  of  the  scene,  they  still  need  to  work 
within  the  sensitivities  of  local  political  structures  and  be  aware  of  delicate  community 
relationships. 

Behavioral  indicators:  You  can  expect  someone  who  is  effectively  establishing  a  chain  of 
command  to: 

•  Clearly  communicate  the  command  structure. 
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•  Know  when  to  relinquish  command  to  the  agency  more  suited  to  deal  with  the  current 
situation. 

•  Empower  staff  to  make  decisions  and  not  dictate  how  to  complete  each  task  or  micro¬ 
manage  Insure  that  only  one  person  is  in  charge. 

•  Clearly  express  intent  and  mission  objectives. 

•  Provide  only  the  necessary  infonnation  to  higher  commanders  and  community  leaders 

Instructor  example:  To  teach  students  to  establish  chain-of-command,  instructors  can  create  an 
inject  where  a  staff  member  knocks  on  the  student’s  door  and  asks  which  person  in  the  EOC  can 
offer  guidance  and  approve  a  plan.  The  student  should  be  able  to  guide  the  staff  member  to  the 
correct  person  or  address  the  staff  member’s  issue  if  it  is  appropriate  for  the  role. 

Execute 

1.  Recognize  Decision  Points. 

Definition:  Effective  commanders  and  staff  recognize  when  in  a  crisis  they  need  to  make  a 
decision.  They  develop  effective  courses  of  action  and  predict  the  effects  of  their  decisions. 

They  continually  evaluate  situations  and  recognize  when  situational  changes  require  changes  in 
action.  They  detennine  when  it  is  appropriate  to  act  versus  wait.  They  plan  for  future  events  and 
take  proactive  steps  to  shape  events  in  their  favor. 

Explanation:  The  decision  to  act  versus  to  wait  requires  a  quick  cost/benefit  analysis,  often 
when  there  are  a  great  number  of  unknowns  facing  the  decision  maker.  Those  with  a  lot  of 
experience  are  able  to  predict  the  outcomes  of  action  (or  inaction).  They  know  when  it  is  time  to 
act  and  when  it  is  okay  to  wait  and  gather  more  infonnation.  They  are  confident;  thus  will  take 
action  in  situations  where  less  experienced  decision  makers  would  wait.  They  can  foresee  how 
the  situation  will  change  based  on  the  actions  they  take.  Because  they  can  predict  the 
consequences  of  their  actions,  they  are  able  to  take  steps  during  an  incident  to  shape  events  in 
their  favor. 

Behavioral  indicators:  You  can  expect  someone  who  effectively  recognizes  decision  points  to: 

•  Take  decisive  action  before  it  is  too  late. 

•  Not  make  hasty  decisions  in  reaction  to  small  changes  in  the  situation. 

•  Recognize  when  a  situation  is  not  an  immediate  threat. 

•  Reacts  to  critical  situations  immediately,  without  taking  time  to  consider  all 
consequences. 

•  Ask  for  outside  assistance  when  they  cannot  handle  a  situation  without  outside 
assistance. 

•  Willing  to  change  action  in  response  to  unfolding  events. 

•  Notices  changes  in  conditions  and  when  to  shift  priorities. 

•  Take  effective  action  when  plans  do  not  go  as  expected. 

•  Be  able  to  conduct  current  operations  while  planning  future  operations. 

•  Be  able  to  slow  down  and  observe  the  situation  instead  of  rushing  in. 

•  Be  able  to  adjust  procedures  to  fit  the  current  situation. 
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Instructor  example:  To  teach  students  to  recognize  decision  points,  the  instructor  can  create  an 
ambiguous  situation  that  requires  information  from  different  sources,  such  as  conflicting  reports 
of  explosions  (i.e.,  somebody  says  it’s  a  bomb  threat,  someone  else  says  it’s  a  bomb  explosion,  a 
third  person  says  it’s  the  explosion  of  a  gas  main).  The  student  should  gather  enough 
infonnation  to  understand  the  correct  situation  and  take  action.  For  instance,  the  student  receives 
more  reports  of  a  bomb  explosion,  receives  Intel  of  a  terrorist  threat,  and  gets  a  call  from  the 
incident  commander  who  says  it  is  a  bomb  explosion.  The  student  should  take  actions  to  secure 
Post,  evacuate  people,  and  organize  a  response. 

2.  Address  Public  and  Responder  Needs. 

Definition:  Effective  commanders  and  staff  notify  the  public  and  media  about  the  situation. 

They  use  a  variety  of  media  platforms  to  inform  the  public  about  upcoming  responder  actions. 
They  are  sensitive  to  community  needs  and  act  to  alleviate  any  public  fear  and/or  resentment. 
They  attend  to  the  personal  and  professional  needs  of  the  responders. 

Explanation:  It  is  necessary  to  work  with  the  media  to  disseminate  accurate  infonnation  and 
calm  public  fear.  Good  infonnation  management  prevents  erroneous  reports  and  negative 
rumors  about  responder  actions  and  intent.  If  responders  do  not  communicate  with  the  media, 
the  media  will  release  stories  based  on  the  unauthorized  reports  of  witnesses  and  bystanders. 
Decision  makers  need  to  release  a  message  that  is  accurate  and  sensitive  to  the  needs  and 
concerns  of  the  community.  They  also  need  to  be  responsive  to  public  need,  rather  than  appear 
insensitive.  It  is  vital  that  decision  makers  attend  to  the  needs  of  responders,  who  need  to 
maintain  energy  and  good  health.  Responders  to  local  disasters  are  concerned  about  their 
families  and  need  leeway  to  care  for  loved  ones.  To  keep  morale  up,  decision  makers  need  to 
show  empathy  for  responders’  concerns  for  their  families.  Decision  makers  should  make 
accommodations  for  responders  who  have  obligations  to  other  employers. 

Behavioral  indicators:  You  can  expect  someone  who  effectively  recognizes  decision  points  to: 

•  Take  proactive  steps  to  prevent  casualties. 

•  Determine  if  contractors  should  handle  situation  before  acting  (i.e.,  with  bulldozer). 

•  Communicate  with  the  public  about  actions  the  military  is  going  to  take. 

•  Allow  responders/troops  to  deal  with  family  and  employment  issues. 

•  Show  concern  for  public  frustration  and  fear. 

•  Use  the  media  to  get  infonnation  out  to  the  community. 

•  Insure  that  responders  get  food  and  sleep. 

Instructor  example:  To  teach  students  to  address  public  and  responder  needs,  the  instructor  can 
create  a  telephone  call  from  the  media  asking  to  confinn  reports  of  an  incident  on  Post  that  is 
unknown  to  the  student,  such  as  an  airplane  crash.  The  student  should  confinn  the  report  and 
take  steps  to  control  the  news  story  in  a  way  that  insure  only  conect  information  is  reported,  and 
only  when  the  time  is  right. 
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3.  Maintain  Mission  Priorities 


Definition:  Effective  commanders  and  staff  recognize  that  their  primary  mission  is  to  protect 
human  life  and  property  while  insuring  the  safety  of  incident  responders.  They  resist  responding 
to  missions  or  requests  that  do  not  advance  mission  goals  or  that  neglect  mission  priorities.  They 
do  not  lose  sight  of  the  mission  priorities  when  dealing  with  external  pressures  or  personal 
disputes. 

Explanation:  It  is  important  to  keep  focused  on  the  mission  goals  and  not  be  distracted  by 
secondary  tasks  or  issues  that  are  not  critical  to  the  big  picture.  External  pressures,  such  as 
commanders’  personal  agendas  and  the  priorities  of  political  leaders,  can  distract  decision 
makers  from  mission  priorities.  In  instances  where  the  community  is  suffering  and  people  are 
asking  for  help,  decision  makers  must  determine  what  they  can  and  cannot  support  both  in  terms 
of  available  resources  and  in  terms  of  mission  priorities  and  scope.  The  decision  maker  must 
look  beyond  the  immediate  needs  of  those  in  front  of  them  to  the  long-term  needs  of  the  larger 
population. 

Behavioral  indicators:  You  can  expect  someone  who  effectively  maintains  mission  priorities  to: 

•  Not  try  to  feed  everyone  or  address  needs  outside  mission  scope. 

•  Take  steps  to  insure  safety  rather  than  react  on  emotion. 

•  Takes  actions  that  reflect  mission  priorities. 

•  Act  appropriately  when  in  a  supporting  role. 

•  Use  diplomacy  when  resisting  external  political  pressures. 

•  Mitigate  consequences  of  non-mission  specific  actions  both  in  the  short-  and  long¬ 
term. 

Instructor  example:  To  teach  students  to  maintain  mission  priorities,  the  instructor  can  create  a 
report  from  a  staff  member  stating  that  responders  are  reporting  exhaustion  after  36  hours  into  a 
rescue  operation.  The  same  responders  have  worked  the  entire  36  hours  with  no  relief  or  backup 
available.  Students  should  remedy  the  situation  by  finding  replacement  staff,  and  by  providing 
food,  cots,  showers,  and  other  necessities.  This  situation  can  be  prevented  and  instructors  should 
provide  students  the  opportunity  to  take  preventative  actions  such  as  insuring  a  relief  schedule  is 
set  up  and  manpower  is  available.  If  they  do  not  take  these  actions,  they  should  see  the 
consequences  (exhaustion,  loss  of  manpower  with  no  replacements). 
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Appendix  D 


Evaluation  Survey  Day  1:  Camp  Atterbury 

POST  RUN  EVALUATION  FORM 
PARTICIPANT  REACTION  -  TUF-D  SIMULATION 


Company:  Instructor(s): 

Conducted  by: 

Course:  TUF-D  Simulation  Training  &  Evaluation 

Date: 

Unit/Rank: _ 

Position: _ 

Years  in  this  position: _ 

Role  during  actual  emergency  event: 
Role  Played  in  Simulation: _ 


Five  learning  objectives  are  listed  below.  Please  rate  the  degree  that  your  ability  to  accomplish 
these  objectives  improved  due  to  participation  in  this  simulation.  Use  the  following  scale: 

1  -  None  =  no  apparent  improvement  in  my  ability  to  perform  this  objective 

2  -  Slight  =  slight  improvement  in  my  ability  to  perform  this  objective 

3  -  Moderate  =  moderate  improvement  in  my  ability  to  perform  this  objective 

4  -  Substantial  =  substantial  improvement  in  my  ability  to  perform  this  objective 

5  -  Exceptional  =  exceptional  improvement  in  my  ability  to  perform  this  objective 

After  completing  the  simulation, 
the  participant  will  be  able  to... 

Improvement 

None 

Slight 

Moderate 

Substantial 

Exceptional 

1.  Manage  information  and  communications 
during  and  after  an  earthquake  disaster 

1 

2 

3 

4 

5 

2.  Maintain  focus  on  mission  priorities 

1 

2 

3 

4 

5 

3.  Assess  the  situation  and  select  the  best 
course(s)  of  action 

1 

2 

3 

4 

5 

4.  Reflect  upon  the  simulated  experience  and 
discuss  the  reasons  for  their  decisions 

1 

2 

3 

4 

5 

5.  Identify  and  use  assets  as  needed 

1 

2 

3 

4 

5 
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Please  rate  the  following  comments  about  the  Simulation  using  the  following  scale: 

1  =  Strongly  Disagree  2  =  Disagree  3  =  Neutral  4  =  Agree  5  =  Strongly  Agree 

The  Simulator 

Agreement 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

1.  The  simulation  presented  realistic  content 

1 

2 

3 

4 

5 

2.  I  have  worked  harder  during  this  exercise 
than  in  most  exercises  I  have  participated  in 

1 

2 

3 

4 

5 

3.  From  using  this  simulation  tool,  I  have  a 
better  understanding  of  how  to  make 
decisions  during  a  crisis 

1 

2 

3 

4 

5 

4.  Overall,  I  rate  this  simulation  tool  as 
excellent 

1 

2 

3 

4 

5 

5.  I  would  participate  in  training  using  the 
simulation  in  the  future 

1 

2 

3 

4 

5 

Short  Answer 

Directions:  Please  answer  the  following  questions  about  using  the  simulator. 

1.  Did  the  simulation  tool  meet  your  expectations?  Y  N 
Why  or  why  not 


2.  To  what  degree  did  the  simulation  environment  present  a  realistic  environment  in  which  you 
could  perform  your  assigned  role 


Not  at  all  realistic 

1 

2 

3 

4 

5 

Very  realistic 

Please  provide  an  explanation  of  your  rating: 


3.  How  closely  did  your  role  in  the  simulation  match  your  role  in  the  real  world? 


No  match  at  all 

1 

2 

3 

4 

5 

Exact  match 

Why  or  why  not 
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4.  Should  this  simulation  tool  be  incorporated  into  training/education  courses? 


Do  not  recommend 

1 

2 

3 

4 

5 

Highly  recommend 

Why  or  why  not? 


5.  Please  comment  on  skills,  concepts,  and  techniques  you  learned  by  using  the  simulation  tool: 


6.  What  did  you  like  best  about  using  the  simulator? 


7.  What  did  you  like  least  about  using  the  simulator? 


ADDITIONAL  COMMENTS: 


D-3 


Appendix  E 


Evaluation  Survey  Day  2:  Camp  Atterbury 


POST  RUN  EVALUATION  FORM 
PARTICIPANT  REACTION  -  SCENARIO  DEVELOPER 


Company:  Instructor(s): 

Conducted  by: 

Course:  TUF-D  Vignette  Creation  Tool  Training  &  Evaluation 

Date: 

Unit/Rank: 

Position: 


Years  in  this  position: 


Role  during  actual  emergency  event: 


Role  Played  in  Simulation: 


Five  learning  objectives  are  listed  below.  Please  rate  the  degree  that  your  ability  to  accomplish 
these  objectives  improved  due  to  participation  in  this  simulation.  Use  the  following  scale: 

1  -  None  =  no  apparent  improvement  in  my  ability  to  perform  this  objective 

2  -  Slight  =  slight  improvement  in  my  ability  to  perform  this  objective 

3  -  Moderate  =  moderate  improvement  in  my  ability  to  perform  this  objective 

4  -  Substantial  =  substantial  improvement  in  my  ability  to  perform  this  objective 

5  -  Exceptional  =  exceptional  improvement  in  my  ability  to  perform  this  objective 

After  completing  the  scenario 
developer  training,  the  participant 
will  be  able  to... 

Improvement 

None 

Slight 

Moderate 

Substantial 

Exceptional 

6.  Explain  the  scenario  development  process 

1 

2 

3 

4 

5 

7.  Identify  different  types  of  simulated  injects 

1 

2 

3 

4 

5 

8.  Add,  delete,  and  modify  timeline  segments 

1 

2 

3 

4 

5 

9.  Add,  delete,  and  modify  simulation  injects 

1 

2 

3 

4 

5 

10.  Construct  an  assessment 

1 

2 

3 

4 

5 
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Please  rate  the  following  comments  about  the  Scenario  Developer  using  the  following  scale: 

1  =  Strongly  Disagree  2  =  Disagree  3  =  Neutral  4  =  Agree  5  =  Strongly  Agree 

The  Scenario  Developer 

Agreement 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

6.  The  scenario  developer  was  easy  to  use 

1 

2 

3 

4 

5 

7.  This  tool  will  allow  me  to  create  realistic 
scenarios  and  provide  useful  and  relevant 
training 

1 

2 

3 

4 

5 

8.  From  using  the  scenario  developer  tool,  I 
have  a  better  understanding  of  how  the 
simulation  works 

1 

2 

3 

4 

5 

9.  Overall,  I  rate  the  scenario  developer  tool  as 
excellent 

1 

2 

3 

4 

5 

10.  I  would  use  the  scenario  developer  tool  in 
the  future 

1 

2 

3 

4 

5 

Short  Answer 

Directions:  Please  answer  the  following  questions  about  using  the  simulator. 

8.  Did  the  scenario  developer  tool  meet  your  expectations?  Y  N 
Why  or  why  not. 

9.  Would  you  recommend  the  scenario  developer  tool  to  others?  Y  N 
Why  or  why  not? 


10.  Please  comment  on  skills,  concepts,  and  techniques  you  learned  by  using  the  scenario 
developer  tool: 

11.  What  did  you  like  best  about  using  the  scenario  developer  tool? 


12.  What  did  you  like  least  about  using  the  scenario  developer  tool? 
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